Skip to main content

FeedMail Down

FeedMail is offline. This appears to be due to an outage at DigitalOcean, our primary hosting provider. Their status page says nothing is wrong but their control panel won't even load. We are trying to contact support.

Updates will be posted here as they become available.

All times are in UTC.

Update 23:52

FeedMail appears to be back online. Feeds have been polled and updates sent. DigitalOcean is still having problems. We are monitoring the situation.

Update 00:06

DigitalOcean has published an incident to track this global outage. FeedMail is not currently seeing any direct impact but DigitalOcean still says that there may be elevated error rates.

https://status.digitalocean.com/incidents/xkpsn9hbnnlz

Update 20:57

DigitalOcean claims that they have resolved this issue and any remaining errors should still be subsiding.

FeedMail is still operating normally and it seems that no further disruptions are expected.

Summary

FeedMail experienced two periods of downtime, from June 13th 23:26 - 23:50 and June 14th 02:52 - 03:01. These downtime periods correspond to events from our hosting provider, a global outage and "emergency maintenance [...] to implement a long-term fix" in the datacenter that FeedMail uses. FeedMail automatically resumed operation in both cases without human intervention.

graph of FeedMail operation

This is a graph of FeedMail operations. The spiky behaviour after the outages is checking the missed updates in the outage window, these spikes are rate-limited and slowly smooth out over time.

During these outages feeds were not updated, updates were not sent and the website was unavailable (other than static content that was cached by our CDN). We don't believe that any updates were missed or any data lost in their period, updates were simply delayed.

The root cause of this failure is that FeedMail has dependencies on single provides and single regions. We have no immediate plans to improve this situation as keeping a simple and cost-effective infrastructure is important to us. However as we have more resources available we will consider how availability can be improved.

Comments

Popular posts from this blog

Delivery Delays to Gmail

In the past 48 hours Google has started delaying the delivery of some FeedMail notifications. This is currently affecting about 10% of messages to Gmail users. These notifications will be resent with a delay. We also speculate that some notifications will be marked as spam.   Update : As of 2023-05-09 this appears to be resolved. If You Are Affected If you use Gmail you may be affected by this. Notifications may be delayed or marked as spam. If your notifications are marked as spam you can create a filter to avoid this. Use "from:*@feedmail.org" as the rule and select " Never send it to Spam". If your notifications are delayed we are unaware of any action that you can take. However marking notifications that ended up in your spam folder as "Not Spam" may help avoid future delays.  It does appear that these emails are eventually being accepted but we are unsure if that means that they are actually ending up in users' mailboxes (or even their spam folder

Updates to HTML Processing

Since its inception FeedMail has done processing on HTML content in feeds to ensure that it renders as expected in email form. At first this was fairly simple things like rewriting URLs to point to the correct location (many feeds use non-absolute URLs that won't work in email) but over time more complex transformations were added such as adding fallback content to media embeds without any. The full-text scraping feature requires even more complex processing as it requires stripping away most of the page and handling content that was designed for full-featured browsers. What changed? Recently FeedMail has migrated all HTML rewriting to use new infrastructure. This provides more flexibility and enabled new features (such as showing controls on all media embeds) and made our processing much more reliable. What does this mean to me? As a user you shouldn't see much difference. Overall the emails you receive should be better formatted but the difference will be subtle. Full-text sc

Email Headers

FeedMail now sets some email headers that advanced users can use to filter the messages that FeedMail sends. For example you can filter notifications in a specific category into a different folder. The headers that we set are documented in the FAQ . These headers have been being set on notifications for a few months now so you can use past messages to test your filters. If you have any questions, or would like to see other headers set to help your filtering please let us know .