Skip to main content

DNS Outage

From 2024-08-26 19:46 to 2024-08-27 11:21 UTC FeedMail had an outage.

  • Until 2024-06-26 20:34 FeedMail was completely down.
  • For the remainder of the outage most emails not sent.

It is expected that no feed updates were lost during this outage. Updates would only be lost if they were only present on the feed within the 50min of total outage. Most feeds ensure that updates are present for days so this would not be an issue.

Notifications have been delayed and should be sent by 2024-08-27 12:31. This may take longer if your mail provider applies limits and FeedMail needs to retry delivery at a later time.

Update: All delayed notifications have been sent successfully.

Timeline

All times are in UTC.

2024-08-2619:46StartFeedMail goes down. 

19:53DetectionAutomated monitoring reported that feeds were not being checked.

20:34
The Database IP was hardcoded, restoring most functionality.
2024-08-2711:21ResolutionFeedMail was switched external DNS.

11:24
Schedule of delayed mail was adjusted to send in the next hour.

Analysis

This outage was triggered by the default DNS of our hosting provider failing. This resulted in the failure of some key functionality including:

  • Database Access (and anything that depends on it).
  • Monitoring
  • Mail Sending

It is worth noting that feed checking was not directly affected as FeedMail uses it's own DNS resolver for most internal requests. The affected functionality was due to requests were made by libraries and APIs that use the system resolver.

Satabase access as it is required by almost all functionality. This eventually lead to FeedMail failing health checks then being restarted where it failed to start up.

At first some experiments were attempted to resolve the DNS issue.

  • Create a fresh Kubernetes node with our hosting provider and move FeedMail to that node. This node had the same DNS issue.
  • Restart all system Kubernetes pods to see if they came back functioning.

None of these resolved the problem.

Next the database IP was hardcoded in the configuration avoiding the DNS dependency for database access and restoring most functionality. At this point the only missing functionality were:

  • Monitoring
  • Mail Sending

A support ticket was filed with out hosting provider describing the problem, but as of now it hasn't been resolved. But we have been informed that it has been forwarded to their response team.

Next we updated our DNS configuration to avoid the provider's DNS server. This resolved all remaining functionality. 

What Went Well

  • It was easy to hardcode the database IP and restore most functionality.

What Went Poorly

  • Monitoring was taken offline so manual checks needed to be performed. These were unreliable and missed the problem with mail sending.
  • Despite having our own DNS resolver configured it isn't used for everything.
  • It was not immediately identified that mail sending depended on the system DNS.
  • Our hosting provider did not respond promptly to a serious outage.

Action Items

We will move mail-sending to fully utilize our own DNS resolver. It was believed that this was the case as we already look up the MX records using our resolver. But it wasn't noticed that the mail server host names are passed to the OS for resolution. We will ensure that we are resolving the records ourselves to gain the reliability and security benefits of our internal resolver.

Labels:

Comments

Post a Comment

Popular posts from this blog

Delivery Delays to Gmail

In the past 48 hours Google has started delaying the delivery of some FeedMail notifications. This is currently affecting about 10% of messages to Gmail users. These notifications will be resent with a delay. We also speculate that some notifications will be marked as spam.   Update : As of 2023-05-09 this appears to be resolved. If You Are Affected If you use Gmail you may be affected by this. Notifications may be delayed or marked as spam. If your notifications are marked as spam you can create a filter to avoid this. Use "from:*@feedmail.org" as the rule and select " Never send it to Spam". If your notifications are delayed we are unaware of any action that you can take. However marking notifications that ended up in your spam folder as "Not Spam" may help avoid future delays.  It does appear that these emails are eventually being accepted but we are unsure if that means that they are actually ending up in users' mailboxes (or even their spam folder
Post a Comment
Read more

Updates to HTML Processing

Since its inception FeedMail has done processing on HTML content in feeds to ensure that it renders as expected in email form. At first this was fairly simple things like rewriting URLs to point to the correct location (many feeds use non-absolute URLs that won't work in email) but over time more complex transformations were added such as adding fallback content to media embeds without any. The full-text scraping feature requires even more complex processing as it requires stripping away most of the page and handling content that was designed for full-featured browsers. What changed? Recently FeedMail has migrated all HTML rewriting to use new infrastructure. This provides more flexibility and enabled new features (such as showing controls on all media embeds) and made our processing much more reliable. What does this mean to me? As a user you shouldn't see much difference. Overall the emails you receive should be better formatted but the difference will be subtle. Full-text sc
Post a Comment
Read more

Providing Email Subscriptions to your Readers with FeedMail

If you offer a blog with an RSS feed you can reach more users by offering email subscription as well. FeedMail provides an easy-to-integrate newsletter that has no cost to you. Just select one of the implementation strategies below. Implementation Options Link The easiest option is to provide a link to the subscribe page. First got to the FeedMail New Subscription page. Enter your website's URL and click "Go".  Then copy the URL from your browser's address bar. This the the URL to subscribe to your website! It should look something like https://feedmail.org/subscriptions/new?url= https%3A%2F%2Fyour-site.example . You can then link to that URL from your site. For example if you are using raw HTML in your design it would look something like: <a href="https://feedmail.org/subscriptions/new?url= https%3A%2F%2Fyour-site.example ">Click here to subscribe by email.</a>  Subscribe Button To use a button simply add the following code to your website. &l
Post a Comment
Read more