Skip to main content

FeedMail was Down

FeedMail was offline for 26 minutes. During this period the website was unavailable and feed updates were not sent.

This outage was caused by our CoreDNS resolver failing. While FeedMail continued operating normally for a while as most operations such as feed fetching and mail sending don't rely on the Kubernetes DNS server FeedMail does use the Kubernetes DNS server for a few operations such as connecting to it's own database. When database connections needed to be refreshed the DNS resolution failure caused FeedMail to become unhealthy and it was unable to continue operation.

Timeline

All times are in UTC.

13:28StartFeedMail goes down. Website is offline and feeds are not being checked.
13:32DetectionAutomated monitoring reported that the FeedMail website was unavailable.
13:38
Automated monitoring reported that feeds were not being fetched.
13:42
Kubernetes cluster update was started.
13:53MitigatedFeedMail was restored to operation. The website was again available and feeds started being checked.
13:54ResolvedAll feeds were checked and mail was sent.
Note that WebSub updates that fired during the downtime may take slightly longer to appear as the server will select the retry interval.

Analysis

CoreDNS was returning 503 to its readiness healthy check and had the following message repeated in its logs. 

plugin/ready: Still waiting on: "kubernetes"

No changed had recently been made to CoreDNS. Restarting CoreDNS did not help.

This incident was resolved by updating Kubernetes. This update was announced earlier in the day and we were planning on waiting a few days to apply it in case any bugs were found and fixed in the new version. Instead it was decided to apply it immediately to reconfigure CoreDNS or the Kubernetes API server to a working state. This was a risky maneuver but since FeedMail runs on a managed Kubernetes cluster we don't configure CoreDNS ourselves so it seemed safer than manually tweaking settings, especially since the true issue may have been with the Kubernetes API server.

What Went Well

  • Monitoring quickly detected the issue.
  • The service quickly and gracefully recovered once DNS resolution was restored.

What Went Poorly

Nothing.

Where We Got Lucky

  • The Kubernetes update was released only hours before fixed the issue.
    • If it didn't or hadn't been released we would have had to file a service request which likely would have taken longer.

Action Items

At this time we don't except to take any action. This downtime is within our reliability targets. The cost to resolve this issue is not deemed worth it at this time.

One mitigation would be to run multiple Kubernetes clusters. This would give us software version and geographical isolation. However this would increase operational complexity as well as costs. Another option may be to run more instances of CoreDNS but this is managed by our provider so we would prefer not to customize it at this time.

One last option would be to override DNS settings and use our own DNS resolvers for all operations. This is something that we will continue to revisit in the future.

Labels:

Comments

Post a Comment

Popular posts from this blog

Delivery Delays to Gmail

In the past 48 hours Google has started delaying the delivery of some FeedMail notifications. This is currently affecting about 10% of messages to Gmail users. These notifications will be resent with a delay. We also speculate that some notifications will be marked as spam.   Update : As of 2023-05-09 this appears to be resolved. If You Are Affected If you use Gmail you may be affected by this. Notifications may be delayed or marked as spam. If your notifications are marked as spam you can create a filter to avoid this. Use "from:*@feedmail.org" as the rule and select " Never send it to Spam". If your notifications are delayed we are unaware of any action that you can take. However marking notifications that ended up in your spam folder as "Not Spam" may help avoid future delays.  It does appear that these emails are eventually being accepted but we are unsure if that means that they are actually ending up in users' mailboxes (or even their spam folder
Post a Comment
Read more

Updates to HTML Processing

Since its inception FeedMail has done processing on HTML content in feeds to ensure that it renders as expected in email form. At first this was fairly simple things like rewriting URLs to point to the correct location (many feeds use non-absolute URLs that won't work in email) but over time more complex transformations were added such as adding fallback content to media embeds without any. The full-text scraping feature requires even more complex processing as it requires stripping away most of the page and handling content that was designed for full-featured browsers. What changed? Recently FeedMail has migrated all HTML rewriting to use new infrastructure. This provides more flexibility and enabled new features (such as showing controls on all media embeds) and made our processing much more reliable. What does this mean to me? As a user you shouldn't see much difference. Overall the emails you receive should be better formatted but the difference will be subtle. Full-text sc
Post a Comment
Read more

Email Headers

FeedMail now sets some email headers that advanced users can use to filter the messages that FeedMail sends. For example you can filter notifications in a specific category into a different folder. The headers that we set are documented in the FAQ . These headers have been being set on notifications for a few months now so you can use past messages to test your filters. If you have any questions, or would like to see other headers set to help your filtering please let us know .
Post a Comment
Read more