Skip to main content

FeedMail was Down

FeedMail was offline for 26 minutes. During this period the website was unavailable and feed updates were not sent.

This outage was caused by our CoreDNS resolver failing. While FeedMail continued operating normally for a while as most operations such as feed fetching and mail sending don't rely on the Kubernetes DNS server FeedMail does use the Kubernetes DNS server for a few operations such as connecting to it's own database. When database connections needed to be refreshed the DNS resolution failure caused FeedMail to become unhealthy and it was unable to continue operation.

Timeline

All times are in UTC.

13:28StartFeedMail goes down. Website is offline and feeds are not being checked.
13:32DetectionAutomated monitoring reported that the FeedMail website was unavailable.
13:38
Automated monitoring reported that feeds were not being fetched.
13:42
Kubernetes cluster update was started.
13:53MitigatedFeedMail was restored to operation. The website was again available and feeds started being checked.
13:54ResolvedAll feeds were checked and mail was sent.
Note that WebSub updates that fired during the downtime may take slightly longer to appear as the server will select the retry interval.

Analysis

CoreDNS was returning 503 to its readiness healthy check and had the following message repeated in its logs. 

plugin/ready: Still waiting on: "kubernetes"

No changed had recently been made to CoreDNS. Restarting CoreDNS did not help.

This incident was resolved by updating Kubernetes. This update was announced earlier in the day and we were planning on waiting a few days to apply it in case any bugs were found and fixed in the new version. Instead it was decided to apply it immediately to reconfigure CoreDNS or the Kubernetes API server to a working state. This was a risky maneuver but since FeedMail runs on a managed Kubernetes cluster we don't configure CoreDNS ourselves so it seemed safer than manually tweaking settings, especially since the true issue may have been with the Kubernetes API server.

What Went Well

  • Monitoring quickly detected the issue.
  • The service quickly and gracefully recovered once DNS resolution was restored.

What Went Poorly

Nothing.

Where We Got Lucky

  • The Kubernetes update was released only hours before fixed the issue.
    • If it didn't or hadn't been released we would have had to file a service request which likely would have taken longer.

Action Items

At this time we don't except to take any action. This downtime is within our reliability targets. The cost to resolve this issue is not deemed worth it at this time.

One mitigation would be to run multiple Kubernetes clusters. This would give us software version and geographical isolation. However this would increase operational complexity as well as costs. Another option may be to run more instances of CoreDNS but this is managed by our provider so we would prefer not to customize it at this time.

One last option would be to override DNS settings and use our own DNS resolvers for all operations. This is something that we will continue to revisit in the future.

Labels:

Comments

Post a Comment

Popular posts from this blog

Delay on YouTube Feeds

Most YouTube feeds have not sent new notifications since 2023-10-02 14:20 UTC. We will be triggering a manual YouTube feeds over the next hour and all missing notifications will be sent. If you have any missing notifications after 2023-10-05 14:00 UTC please reach out to support. Update 14:00 : All updates have been sent. If you believe that you are still missing updates feel free to reach out. The rest of this post is a technical analysis of the issue. Background This was caused due to recent emergency response to YouTube WebSub notifications. The emergency response was necessary is due to the following factors. YouTube WebSub posts are not spec compliant and do not contain the required information to send notifications. Therefore FeedMail uses these notifications as a "ping" to re-fetch the feed. YouTube often sends notifications before the entries appear in the feed. The exact reason is not known but sometimes entries do not appear for up to an hour after the WebSub push. ...
Post a Comment
Read more

Invalid DKIM Signature for Some Mail

As of 2023-07-26 FeedMail messages sent via AWS SES have an invalid feedmail.org DKIM signature. This may result in messages ending up in your spam folder. This is an ongoing issue and updates will be posted here as they are available. 2023-09-25 We have issue an update that should avoid SES re-formatting signed fields. This is a workaround but should result in valid signatures. Who is affected? This is unlikely to affect most of our users for the following reasons: 97% of our mail is sent by us directly, not via AWS. This mail still has a SPF-verified sending IP. However this may still affect users because spam filters may consider these messages less "good" than they would have been with a valid DKIM signature. The most notable Inbox Providers affected by this are Apple and Microsoft. FeedMail uses AWS SES for these providers as they reject all messages from our network provider. However other smaller providers may also have a portion of their messages sent via AWS SES. ...
Post a Comment
Read more

Support for SMTP MTA Strict Transport Security

FeedMail now supports SMTP MTA Strict Transport Security (MTA-STS) . This standard provides receiving domains a way to attempt to indicate that incoming mail should only be delivered over a secure connection. When FeedMail receives this signal it will refuse to deliver over insecure connections (retrying mail as required). FeedMail does not currently support SMTP TLS Reporting (TLSRPT) . We will be reaching out to any existing customers who's mail may be rejected due to this change.
Post a Comment
Read more