Skip to main content

Notification Delivery Delay

Executive Summary

Between 2022-05-12 19:37 and 2022-05-13 12:06 UTC (a period of 16h 29min) notifications for feed updates were not delivered to most customers. These notifications were delivered between 12:16 and 12:36. No notifications were missed.

Technical Details

There was an attempt to enable DNSSEC for MX record lookups. However the applied configuration resulted in errors for domains without DNSSEC configured. This error was not caught during testing because it was expected that this configuration only validated for domains where it was configured and the haphazardly selected test set happened to only include validated domains.

This error was not noticed until 2022-05-13 12:00. This delay was due to quota exhaustion on FeedMail's primary error monitoring service. This resulted in errors not being reported immediately. Instead they were only detected by manual polling of our error dashboard.

Once the cause was identified the following actions were taken:

  1. All items in the unsent email queue were updated to ensure that they would not fail permanently.
  2. A version of FeedMail with DNSSEC disabled was deployed.
  3. All items in the unsent email queue were manually updated to send over the next 20 minutes.
  4. Logs were checked to identify if any entries were permanently dropped.

Action Items

Error Monitoring Quota Alerts

Right now we lack a good way to identify if we are approaching our primary error alerting quota at a rate that will result in quota depletion before the end of the month.

In this case a dependency update caused a high-frequency warning message which depleted our error quota over a few days early in the month. By the time it is clear that it was necessary to revert that dependency it was too late. Early warning of quota usage would have sufficiently mitigated this problem.

Backup Error Monitoring

Instead of relying on manual polling for backup monitoring an alert could have been set up in our logging framework to notify of this problem rapidly. This likely would have resulted in the error being fixed in 1 hour instead of 16.

This alert has now been set up.

Maintain Test MX List

This was also a preventable error. It was an intentional change to DNSSEC. However it happens that the handful of domains this was tested on all happened to have DNSSEC configured. In order to ensure sufficient testing in the future a list of "Text MXes" will be documented which contains accounts on publicly accessible MXes that exercise a variety of DNS configuration and mail software.

Re-Enable DNSSEC

DNSSEC is critical to the security of email. We will investigate how to restore DNSSEC validation to ensure the security of domains that support it. We are also considering alternate security mechanisms such as PGP encrypted notifications (let us know if you are interested)
As always, if you have any comments, questions or feedback simply reach out to FeedMail Support.
Labels:

Comments

Post a Comment

Popular posts from this blog

Digests are Coming

Up to this point FeedMail has only supported real-time notifications. Meaning that every feed update immediately produces a single email. However this is about to change! When we asked for feedback on the features you would like to see in FeedMail we had a number of users reach out saying that they wanted a way to batch notifications together. We saw two main reasons for this: To reduce noise in their inbox. For some high-volume feeds users wanted to be able to quickly skim, then delete the entire batch in one go. While deleting one-by-one offers more flexibility, the bulk option is easier for high-volume feeds. To reduce costs. While we believe that our prices are incredibly reasonable, they can add up if you are getting lots of updates. For example if you follow a feed that updates every 15min that will be about $35 a year (or half price if you buy your credits in bulk). Not super expensive but maybe more than you want to spend for a single feed! Digests provide and option for cost
Post a Comment
Read more

Digests Leave Beta

Thanks everyone who has helped evaluate digests over the past weeks. All of the blocking issues are now resolved and we will be releasing them soon. Once digests are officially released there will be links to them from the FeedMail site and pricing information added to our homepage. Price Increase Part of the purpose of the beta was to evaluate the cost of providing digests and see how they would be used. We have decided upon final pricing which we hope will be sustainable for years to come. Digests issues will cost 1 credit per 5 feeds. Note that this is feeds included in an issue , not total feeds that target a particular digest. It also does not matter how many new items a feed has. So if you have a digest with 200 feeds configured but this morning's issue only has new items from 2 of them it will cost 1 credit. If 14 feeds update the next day that issue will cost 3 credits. If the day after has no updates it will cost nothing. This new pricing takes effect no earlier than 202
Post a Comment
Read more

Update to Date-based Entry Ignoring

TL;DR FeedMail will now ignore new items 7 days older than a previously seen item. This is expected to affect almost no "true" new posts. In theory checking to find new entries for a feed is a simple process. Download the feed. Check the ID of each entry to see if you have seen it before. However the real world is much messier. It is recommended for feed IDs to be URLs (to ensure global uniqueness) however this results in many feeds just using the URL that the article is available at. However these URLs sometimes change, and poorly designed feed generators update the ID of existing entries to the new URL. From a protocol point of view these are completely new entries, however to a user these are duplicates. In order to reduce the effect of this common issue on our our users FeedMail has some simple mitigations for posts that have recorded published dates. If the entry is older than a year always ignore it. If the entry is older than the 10th newest post in the feed ignore it.
Post a Comment
Read more