FeedMail is offline. This appears to be due to an outage at DigitalOcean, our primary hosting provider. Their status page says nothing is wrong but their control panel won't even load. We are trying to contact support.
Updates will be posted here as they become available.
All times are in UTC.
Update 23:52
FeedMail appears to be back online. Feeds have been polled and updates sent. DigitalOcean is still having problems. We are monitoring the situation.
Update 00:06
DigitalOcean has published an incident to track this global outage. FeedMail is not currently seeing any direct impact but DigitalOcean still says that there may be elevated error rates.
https://status.digitalocean.com/incidents/xkpsn9hbnnlz
Update 20:57
DigitalOcean claims that they have resolved this issue and any remaining errors should still be subsiding.
FeedMail is still operating normally and it seems that no further disruptions are expected.
Summary
FeedMail experienced two periods of downtime, from June 13th 23:26 - 23:50 and June 14th 02:52 - 03:01. These downtime periods correspond to events from our hosting provider, a global outage and "emergency maintenance [...] to implement a long-term fix" in the datacenter that FeedMail uses. FeedMail automatically resumed operation in both cases without human intervention.
This is a graph of FeedMail operations. The spiky behaviour after the outages is checking the missed updates in the outage window, these spikes are rate-limited and slowly smooth out over time.
During these outages feeds were not updated, updates were not sent and the website was unavailable (other than static content that was cached by our CDN). We don't believe that any updates were missed or any data lost in their period, updates were simply delayed.
The root cause of this failure is that FeedMail has dependencies on single provides and single regions. We have no immediate plans to improve this situation as keeping a simple and cost-effective infrastructure is important to us. However as we have more resources available we will consider how availability can be improved.
Comments
Post a Comment