Skip to main content

FeedMail Down

FeedMail is offline. This appears to be due to an outage at DigitalOcean, our primary hosting provider. Their status page says nothing is wrong but their control panel won't even load. We are trying to contact support.

Updates will be posted here as they become available.

All times are in UTC.

Update 23:52

FeedMail appears to be back online. Feeds have been polled and updates sent. DigitalOcean is still having problems. We are monitoring the situation.

Update 00:06

DigitalOcean has published an incident to track this global outage. FeedMail is not currently seeing any direct impact but DigitalOcean still says that there may be elevated error rates.

https://status.digitalocean.com/incidents/xkpsn9hbnnlz

Update 20:57

DigitalOcean claims that they have resolved this issue and any remaining errors should still be subsiding.

FeedMail is still operating normally and it seems that no further disruptions are expected.

Summary

FeedMail experienced two periods of downtime, from June 13th 23:26 - 23:50 and June 14th 02:52 - 03:01. These downtime periods correspond to events from our hosting provider, a global outage and "emergency maintenance [...] to implement a long-term fix" in the datacenter that FeedMail uses. FeedMail automatically resumed operation in both cases without human intervention.

graph of FeedMail operation

This is a graph of FeedMail operations. The spiky behaviour after the outages is checking the missed updates in the outage window, these spikes are rate-limited and slowly smooth out over time.

During these outages feeds were not updated, updates were not sent and the website was unavailable (other than static content that was cached by our CDN). We don't believe that any updates were missed or any data lost in their period, updates were simply delayed.

The root cause of this failure is that FeedMail has dependencies on single provides and single regions. We have no immediate plans to improve this situation as keeping a simple and cost-effective infrastructure is important to us. However as we have more resources available we will consider how availability can be improved.

Labels:

Comments

Post a Comment

Popular posts from this blog

Delay on YouTube Feeds

Most YouTube feeds have not sent new notifications since 2023-10-02 14:20 UTC. We will be triggering a manual YouTube feeds over the next hour and all missing notifications will be sent. If you have any missing notifications after 2023-10-05 14:00 UTC please reach out to support. Update 14:00 : All updates have been sent. If you believe that you are still missing updates feel free to reach out. The rest of this post is a technical analysis of the issue. Background This was caused due to recent emergency response to YouTube WebSub notifications. The emergency response was necessary is due to the following factors. YouTube WebSub posts are not spec compliant and do not contain the required information to send notifications. Therefore FeedMail uses these notifications as a "ping" to re-fetch the feed. YouTube often sends notifications before the entries appear in the feed. The exact reason is not known but sometimes entries do not appear for up to an hour after the WebSub push. ...
Post a Comment
Read more

Invalid DKIM Signature for Some Mail

As of 2023-07-26 FeedMail messages sent via AWS SES have an invalid feedmail.org DKIM signature. This may result in messages ending up in your spam folder. This is an ongoing issue and updates will be posted here as they are available. 2023-09-25 We have issue an update that should avoid SES re-formatting signed fields. This is a workaround but should result in valid signatures. Who is affected? This is unlikely to affect most of our users for the following reasons: 97% of our mail is sent by us directly, not via AWS. This mail still has a SPF-verified sending IP. However this may still affect users because spam filters may consider these messages less "good" than they would have been with a valid DKIM signature. The most notable Inbox Providers affected by this are Apple and Microsoft. FeedMail uses AWS SES for these providers as they reject all messages from our network provider. However other smaller providers may also have a portion of their messages sent via AWS SES. ...
Post a Comment
Read more

Support for SMTP MTA Strict Transport Security

FeedMail now supports SMTP MTA Strict Transport Security (MTA-STS) . This standard provides receiving domains a way to attempt to indicate that incoming mail should only be delivered over a secure connection. When FeedMail receives this signal it will refuse to deliver over insecure connections (retrying mail as required). FeedMail does not currently support SMTP TLS Reporting (TLSRPT) . We will be reaching out to any existing customers who's mail may be rejected due to this change.
Post a Comment
Read more