Skip to main content

DNS Outage

From 2024-08-26 19:46 to 2024-08-27 11:21 UTC FeedMail had an outage.

  • Until 2024-06-26 20:34 FeedMail was completely down.
  • For the remainder of the outage most emails not sent.

It is expected that no feed updates were lost during this outage. Updates would only be lost if they were only present on the feed within the 50min of total outage. Most feeds ensure that updates are present for days so this would not be an issue.

Notifications have been delayed and should be sent by 2024-08-27 12:31. This may take longer if your mail provider applies limits and FeedMail needs to retry delivery at a later time.

Update: All delayed notifications have been sent successfully.

Timeline

All times are in UTC.

2024-08-2619:46StartFeedMail goes down. 

19:53DetectionAutomated monitoring reported that feeds were not being checked.

20:34
The Database IP was hardcoded, restoring most functionality.
2024-08-2711:21ResolutionFeedMail was switched external DNS.

11:24
Schedule of delayed mail was adjusted to send in the next hour.

Analysis

This outage was triggered by the default DNS of our hosting provider failing. This resulted in the failure of some key functionality including:

  • Database Access (and anything that depends on it).
  • Monitoring
  • Mail Sending

It is worth noting that feed checking was not directly affected as FeedMail uses it's own DNS resolver for most internal requests. The affected functionality was due to requests were made by libraries and APIs that use the system resolver.

Satabase access as it is required by almost all functionality. This eventually lead to FeedMail failing health checks then being restarted where it failed to start up.

At first some experiments were attempted to resolve the DNS issue.

  • Create a fresh Kubernetes node with our hosting provider and move FeedMail to that node. This node had the same DNS issue.
  • Restart all system Kubernetes pods to see if they came back functioning.

None of these resolved the problem.

Next the database IP was hardcoded in the configuration avoiding the DNS dependency for database access and restoring most functionality. At this point the only missing functionality were:

  • Monitoring
  • Mail Sending

A support ticket was filed with out hosting provider describing the problem, but as of now it hasn't been resolved. But we have been informed that it has been forwarded to their response team.

Next we updated our DNS configuration to avoid the provider's DNS server. This resolved all remaining functionality. 

What Went Well

  • It was easy to hardcode the database IP and restore most functionality.

What Went Poorly

  • Monitoring was taken offline so manual checks needed to be performed. These were unreliable and missed the problem with mail sending.
  • Despite having our own DNS resolver configured it isn't used for everything.
  • It was not immediately identified that mail sending depended on the system DNS.
  • Our hosting provider did not respond promptly to a serious outage.

Action Items

We will move mail-sending to fully utilize our own DNS resolver. It was believed that this was the case as we already look up the MX records using our resolver. But it wasn't noticed that the mail server host names are passed to the OS for resolution. We will ensure that we are resolving the records ourselves to gain the reliability and security benefits of our internal resolver.

Labels:

Comments

Post a Comment

Popular posts from this blog

Delay on YouTube Feeds

Most YouTube feeds have not sent new notifications since 2023-10-02 14:20 UTC. We will be triggering a manual YouTube feeds over the next hour and all missing notifications will be sent. If you have any missing notifications after 2023-10-05 14:00 UTC please reach out to support. Update 14:00 : All updates have been sent. If you believe that you are still missing updates feel free to reach out. The rest of this post is a technical analysis of the issue. Background This was caused due to recent emergency response to YouTube WebSub notifications. The emergency response was necessary is due to the following factors. YouTube WebSub posts are not spec compliant and do not contain the required information to send notifications. Therefore FeedMail uses these notifications as a "ping" to re-fetch the feed. YouTube often sends notifications before the entries appear in the feed. The exact reason is not known but sometimes entries do not appear for up to an hour after the WebSub push. ...
Post a Comment
Read more

Invalid DKIM Signature for Some Mail

As of 2023-07-26 FeedMail messages sent via AWS SES have an invalid feedmail.org DKIM signature. This may result in messages ending up in your spam folder. This is an ongoing issue and updates will be posted here as they are available. 2023-09-25 We have issue an update that should avoid SES re-formatting signed fields. This is a workaround but should result in valid signatures. Who is affected? This is unlikely to affect most of our users for the following reasons: 97% of our mail is sent by us directly, not via AWS. This mail still has a SPF-verified sending IP. However this may still affect users because spam filters may consider these messages less "good" than they would have been with a valid DKIM signature. The most notable Inbox Providers affected by this are Apple and Microsoft. FeedMail uses AWS SES for these providers as they reject all messages from our network provider. However other smaller providers may also have a portion of their messages sent via AWS SES. ...
Post a Comment
Read more

Support for SMTP MTA Strict Transport Security

FeedMail now supports SMTP MTA Strict Transport Security (MTA-STS) . This standard provides receiving domains a way to attempt to indicate that incoming mail should only be delivered over a secure connection. When FeedMail receives this signal it will refuse to deliver over insecure connections (retrying mail as required). FeedMail does not currently support SMTP TLS Reporting (TLSRPT) . We will be reaching out to any existing customers who's mail may be rejected due to this change.
Post a Comment
Read more