Skip to main content

Notification Delivery Delay

Executive Summary

Between 2022-05-12 19:37 and 2022-05-13 12:06 UTC (a period of 16h 29min) notifications for feed updates were not delivered to most customers. These notifications were delivered between 12:16 and 12:36. No notifications were missed.

Technical Details

There was an attempt to enable DNSSEC for MX record lookups. However the applied configuration resulted in errors for domains without DNSSEC configured. This error was not caught during testing because it was expected that this configuration only validated for domains where it was configured and the haphazardly selected test set happened to only include validated domains.

This error was not noticed until 2022-05-13 12:00. This delay was due to quota exhaustion on FeedMail's primary error monitoring service. This resulted in errors not being reported immediately. Instead they were only detected by manual polling of our error dashboard.

Once the cause was identified the following actions were taken:

  1. All items in the unsent email queue were updated to ensure that they would not fail permanently.
  2. A version of FeedMail with DNSSEC disabled was deployed.
  3. All items in the unsent email queue were manually updated to send over the next 20 minutes.
  4. Logs were checked to identify if any entries were permanently dropped.

Action Items

Error Monitoring Quota Alerts

Right now we lack a good way to identify if we are approaching our primary error alerting quota at a rate that will result in quota depletion before the end of the month.

In this case a dependency update caused a high-frequency warning message which depleted our error quota over a few days early in the month. By the time it is clear that it was necessary to revert that dependency it was too late. Early warning of quota usage would have sufficiently mitigated this problem.

Backup Error Monitoring

Instead of relying on manual polling for backup monitoring an alert could have been set up in our logging framework to notify of this problem rapidly. This likely would have resulted in the error being fixed in 1 hour instead of 16.

This alert has now been set up.

Maintain Test MX List

This was also a preventable error. It was an intentional change to DNSSEC. However it happens that the handful of domains this was tested on all happened to have DNSSEC configured. In order to ensure sufficient testing in the future a list of "Text MXes" will be documented which contains accounts on publicly accessible MXes that exercise a variety of DNS configuration and mail software.

Re-Enable DNSSEC

DNSSEC is critical to the security of email. We will investigate how to restore DNSSEC validation to ensure the security of domains that support it. We are also considering alternate security mechanisms such as PGP encrypted notifications (let us know if you are interested)

As always, if you have any comments, questions or feedback simply reach out to FeedMail Support.

Comments

Popular posts from this blog

Announcing FeedMail

I'm pleased to be sharing a project that I have been working on for a while and have been thinking about doing for even longer. FeedMail is a simple service that aims to get updates from your favourite websites to your email with no fuss and no nonsense. If you are already sold and want to follow some feeds simply go to feedmail.org to get started. How FeedMail Works FeedMail works using a set of technologies informally called RSS. FeedMail actually supports a variety of feed formats including Atom, RSS2 and RSS1. These feeds are created by websites and updated whenever new content is posted. FeedMail subscribes to these feeds on your behalf and forwards new entries to the email address of your choice. Many websites support these feeds. Just post the URL to an article or website that you want to subscribe to and FeedMail will show you the available feeds. For example the following websites support RSS: YouTube Channels Medium GitHub Releases Tumblr Many news sites Many more... RSS

How to Subscribe to YouTube via Email

YouTube disabled native email notifications in August 2020 . Luckily FeedMail lets you get these notifications back, and they are better than ever! Why Email Notifications? At FeedMail were are a huge fan of email notifications! Synced across all of your devices. Easy to sort and filter. Very customizable. You can decide if you want them to buzz your phone, ding your computer, both, or quietly file into a folder for you to check when you have some downtime. One place for all of your updates. No need to remember to log into YouTube, Odysee, Vimeo, PeerTube and Nebula on every device. Better than Native? We think so! No need to create a YouTube account. Per-video notifications, no batching to mess up your filters and workflow. You can pick a different email address for each channel for easy filtering. Not everything needs to go to your primary address. Instructions Find the channel that you want to subscribe to. If you are watching a video you can get to the channel but clicking the cha

Providing Email Subscriptions to your Readers with FeedMail

If you offer a blog with an RSS feed you can reach more users by offering email subscription as well. FeedMail provides an easy-to-integrate newsletter that has no cost to you. Just select one of the implementation strategies below. Implementation Options Link The easiest option is to provide a link to the subscribe page. First got to the FeedMail New Subscription page. Enter your website's URL and click "Go".  Then copy the URL from your browser's address bar. This the the URL to subscribe to your website! It should look something like https://feedmail.org/subscriptions/new?url= https%3A%2F%2Fyour-site.example . You can then link to that URL from your site. For example if you are using raw HTML in your design it would look something like: <a href="https://feedmail.org/subscriptions/new?url= https%3A%2F%2Fyour-site.example ">Click here to subscribe by email.</a>  Subscribe Button To use a button simply add the following code to your website. &l