Skip to main content

FeedMail Down

FeedMail is offline. This appears to be due to an outage at DigitalOcean, our primary hosting provider. Their status page says nothing is wrong but their control panel won't even load. We are trying to contact support.

Updates will be posted here as they become available.

All times are in UTC.

Update 23:52

FeedMail appears to be back online. Feeds have been polled and updates sent. DigitalOcean is still having problems. We are monitoring the situation.

Update 00:06

DigitalOcean has published an incident to track this global outage. FeedMail is not currently seeing any direct impact but DigitalOcean still says that there may be elevated error rates.

https://status.digitalocean.com/incidents/xkpsn9hbnnlz

Update 20:57

DigitalOcean claims that they have resolved this issue and any remaining errors should still be subsiding.

FeedMail is still operating normally and it seems that no further disruptions are expected.

Summary

FeedMail experienced two periods of downtime, from June 13th 23:26 - 23:50 and June 14th 02:52 - 03:01. These downtime periods correspond to events from our hosting provider, a global outage and "emergency maintenance [...] to implement a long-term fix" in the datacenter that FeedMail uses. FeedMail automatically resumed operation in both cases without human intervention.

graph of FeedMail operation

This is a graph of FeedMail operations. The spiky behaviour after the outages is checking the missed updates in the outage window, these spikes are rate-limited and slowly smooth out over time.

During these outages feeds were not updated, updates were not sent and the website was unavailable (other than static content that was cached by our CDN). We don't believe that any updates were missed or any data lost in their period, updates were simply delayed.

The root cause of this failure is that FeedMail has dependencies on single provides and single regions. We have no immediate plans to improve this situation as keeping a simple and cost-effective infrastructure is important to us. However as we have more resources available we will consider how availability can be improved.

Labels:

Comments

Post a Comment

Popular posts from this blog

Digests are Coming

Up to this point FeedMail has only supported real-time notifications. Meaning that every feed update immediately produces a single email. However this is about to change! When we asked for feedback on the features you would like to see in FeedMail we had a number of users reach out saying that they wanted a way to batch notifications together. We saw two main reasons for this: To reduce noise in their inbox. For some high-volume feeds users wanted to be able to quickly skim, then delete the entire batch in one go. While deleting one-by-one offers more flexibility, the bulk option is easier for high-volume feeds. To reduce costs. While we believe that our prices are incredibly reasonable, they can add up if you are getting lots of updates. For example if you follow a feed that updates every 15min that will be about $35 a year (or half price if you buy your credits in bulk). Not super expensive but maybe more than you want to spend for a single feed! Digests provide and option for cost
Post a Comment
Read more

Digests Leave Beta

Thanks everyone who has helped evaluate digests over the past weeks. All of the blocking issues are now resolved and we will be releasing them soon. Once digests are officially released there will be links to them from the FeedMail site and pricing information added to our homepage. Price Increase Part of the purpose of the beta was to evaluate the cost of providing digests and see how they would be used. We have decided upon final pricing which we hope will be sustainable for years to come. Digests issues will cost 1 credit per 5 feeds. Note that this is feeds included in an issue , not total feeds that target a particular digest. It also does not matter how many new items a feed has. So if you have a digest with 200 feeds configured but this morning's issue only has new items from 2 of them it will cost 1 credit. If 14 feeds update the next day that issue will cost 3 credits. If the day after has no updates it will cost nothing. This new pricing takes effect no earlier than 202
Post a Comment
Read more

Update to Date-based Entry Ignoring

TL;DR FeedMail will now ignore new items 7 days older than a previously seen item. This is expected to affect almost no "true" new posts. In theory checking to find new entries for a feed is a simple process. Download the feed. Check the ID of each entry to see if you have seen it before. However the real world is much messier. It is recommended for feed IDs to be URLs (to ensure global uniqueness) however this results in many feeds just using the URL that the article is available at. However these URLs sometimes change, and poorly designed feed generators update the ID of existing entries to the new URL. From a protocol point of view these are completely new entries, however to a user these are duplicates. In order to reduce the effect of this common issue on our our users FeedMail has some simple mitigations for posts that have recorded published dates. If the entry is older than a year always ignore it. If the entry is older than the 10th newest post in the feed ignore it.
Post a Comment
Read more