Skip to main content

DNS Outage

From 2024-08-26 19:46 to 2024-08-27 11:21 UTC FeedMail had an outage.

  • Until 2024-06-26 20:34 FeedMail was completely down.
  • For the remainder of the outage most emails not sent.

It is expected that no feed updates were lost during this outage. Updates would only be lost if they were only present on the feed within the 50min of total outage. Most feeds ensure that updates are present for days so this would not be an issue.

Notifications have been delayed and should be sent by 2024-08-27 12:31. This may take longer if your mail provider applies limits and FeedMail needs to retry delivery at a later time.

Update: All delayed notifications have been sent successfully.

Timeline

All times are in UTC.

2024-08-2619:46StartFeedMail goes down. 

19:53DetectionAutomated monitoring reported that feeds were not being checked.

20:34
The Database IP was hardcoded, restoring most functionality.
2024-08-2711:21ResolutionFeedMail was switched external DNS.

11:24
Schedule of delayed mail was adjusted to send in the next hour.

Analysis

This outage was triggered by the default DNS of our hosting provider failing. This resulted in the failure of some key functionality including:

  • Database Access (and anything that depends on it).
  • Monitoring
  • Mail Sending

It is worth noting that feed checking was not directly affected as FeedMail uses it's own DNS resolver for most internal requests. The affected functionality was due to requests were made by libraries and APIs that use the system resolver.

Satabase access as it is required by almost all functionality. This eventually lead to FeedMail failing health checks then being restarted where it failed to start up.

At first some experiments were attempted to resolve the DNS issue.

  • Create a fresh Kubernetes node with our hosting provider and move FeedMail to that node. This node had the same DNS issue.
  • Restart all system Kubernetes pods to see if they came back functioning.

None of these resolved the problem.

Next the database IP was hardcoded in the configuration avoiding the DNS dependency for database access and restoring most functionality. At this point the only missing functionality were:

  • Monitoring
  • Mail Sending

A support ticket was filed with out hosting provider describing the problem, but as of now it hasn't been resolved. But we have been informed that it has been forwarded to their response team.

Next we updated our DNS configuration to avoid the provider's DNS server. This resolved all remaining functionality. 

What Went Well

  • It was easy to hardcode the database IP and restore most functionality.

What Went Poorly

  • Monitoring was taken offline so manual checks needed to be performed. These were unreliable and missed the problem with mail sending.
  • Despite having our own DNS resolver configured it isn't used for everything.
  • It was not immediately identified that mail sending depended on the system DNS.
  • Our hosting provider did not respond promptly to a serious outage.

Action Items

We will move mail-sending to fully utilize our own DNS resolver. It was believed that this was the case as we already look up the MX records using our resolver. But it wasn't noticed that the mail server host names are passed to the OS for resolution. We will ensure that we are resolving the records ourselves to gain the reliability and security benefits of our internal resolver.

Labels:

Comments

Post a Comment

Popular posts from this blog

Digests are now Supported for Owner-Paid Feeds

Owner-paid feeds allow feed publishers to provide FeedMail to their subscribers at no cost. For example the FeedMail Blog is an owner-paid feed. Up until now digest subscriptions were not covered by owner-paid plans. Subscribers could select a digest but they would have to pay for the subscriptions themselves. Digests are now fully supported under owner-paid plans. For users: The owner-paid feeds in your digests no longer count towards the cost of the digest. For publishers: Users will now be able to receive your feed as a digest or included in one of their existing digests. You will be charged one credit for each digest issue containing items from your feed (no matter how many items from your feed are in that issue). Notably this cost will never be more than real-time subscriptions would be.
Post a Comment
Read more

Changes to Billing for Multiple Simultaneous Updates

This change may result in increased credit usage for non-digest subscriptions. Am I Affected? Digest Subscriptions are unaffected . Owner-paid Feeds are unaffected . Frequently updating real-time subscriptions are affected . If you have questions regarding your specific subscriptions please contact FeedMail support . Background FeedMail pricing has always been 1 credit per email notification. However in practice FeedMail would charge at most 1 credit every time a feed was checked, even if there were multiple new items on that feed and multiple emails were sent. In other words FeedMail sometimes undercharged users. Going forward, this billing error will be corrected, which may result in higher credit usage for feeds that were previously undercharged. Correcting this issue is essential to preventing abuse, ensuring long-term sustainability, and maintaining fair pricing. Multiple updates are a rare occurrence. It usually happens on on feeds that update very often (faster than once every 5...
Post a Comment
Read more