Skip to main content

FeedMail Down

FeedMail is offline. This appears to be due to an outage at DigitalOcean, our primary hosting provider. Their status page says nothing is wrong but their control panel won't even load. We are trying to contact support.

Updates will be posted here as they become available.

All times are in UTC.

Update 23:52

FeedMail appears to be back online. Feeds have been polled and updates sent. DigitalOcean is still having problems. We are monitoring the situation.

Update 00:06

DigitalOcean has published an incident to track this global outage. FeedMail is not currently seeing any direct impact but DigitalOcean still says that there may be elevated error rates.

https://status.digitalocean.com/incidents/xkpsn9hbnnlz

Update 20:57

DigitalOcean claims that they have resolved this issue and any remaining errors should still be subsiding.

FeedMail is still operating normally and it seems that no further disruptions are expected.

Summary

FeedMail experienced two periods of downtime, from June 13th 23:26 - 23:50 and June 14th 02:52 - 03:01. These downtime periods correspond to events from our hosting provider, a global outage and "emergency maintenance [...] to implement a long-term fix" in the datacenter that FeedMail uses. FeedMail automatically resumed operation in both cases without human intervention.

graph of FeedMail operation

This is a graph of FeedMail operations. The spiky behaviour after the outages is checking the missed updates in the outage window, these spikes are rate-limited and slowly smooth out over time.

During these outages feeds were not updated, updates were not sent and the website was unavailable (other than static content that was cached by our CDN). We don't believe that any updates were missed or any data lost in their period, updates were simply delayed.

The root cause of this failure is that FeedMail has dependencies on single provides and single regions. We have no immediate plans to improve this situation as keeping a simple and cost-effective infrastructure is important to us. However as we have more resources available we will consider how availability can be improved.

Labels:

Comments

Post a Comment

Popular posts from this blog

DNS Outage

From 2024-08-26 19:46 to 2024-08-27 11:21 UTC FeedMail had an outage. Until 2024-06-26 20:34 FeedMail was completely down. For the remainder of the outage most emails not sent. It is expected that no feed updates were lost during this outage. Updates would only be lost if they were only present on the feed within the 50min of total outage. Most feeds ensure that updates are present for days so this would not be an issue. Notifications have been delayed and should be sent by 2024-08-27 12:31. This may take longer if your mail provider applies limits and FeedMail needs to retry delivery at a later time. Update : All delayed notifications have been sent successfully. Timeline All times are in UTC . 2024-08-26 19:46 Start FeedMail goes down.   19:53 Detection Automated monitoring reported that feeds were not being checked. 20:34 The Database IP was hardcoded, restoring most functionality. 2024-08-27 11:21 Resolution FeedMail was switched external DNS. 11:24 Schedule of ...
Post a Comment
Read more

Digests are now Supported for Owner-Paid Feeds

Owner-paid feeds allow feed publishers to provide FeedMail to their subscribers at no cost. For example the FeedMail Blog is an owner-paid feed. Up until now digest subscriptions were not covered by owner-paid plans. Subscribers could select a digest but they would have to pay for the subscriptions themselves. Digests are now fully supported under owner-paid plans. For users: The owner-paid feeds in your digests no longer count towards the cost of the digest. For publishers: Users will now be able to receive your feed as a digest or included in one of their existing digests. You will be charged one credit for each digest issue containing items from your feed (no matter how many items from your feed are in that issue). Notably this cost will never be more than real-time subscriptions would be.
Post a Comment
Read more

Changes to Billing for Multiple Simultaneous Updates

This change may result in increased credit usage for non-digest subscriptions. Am I Affected? Digest Subscriptions are unaffected . Owner-paid Feeds are unaffected . Frequently updating real-time subscriptions are affected . If you have questions regarding your specific subscriptions please contact FeedMail support . Background FeedMail pricing has always been 1 credit per email notification. However in practice FeedMail would charge at most 1 credit every time a feed was checked, even if there were multiple new items on that feed and multiple emails were sent. In other words FeedMail sometimes undercharged users. Going forward, this billing error will be corrected, which may result in higher credit usage for feeds that were previously undercharged. Correcting this issue is essential to preventing abuse, ensuring long-term sustainability, and maintaining fair pricing. Multiple updates are a rare occurrence. It usually happens on on feeds that update very often (faster than once every 5...
Post a Comment
Read more