Skip to main content

Notification Delivery Delay

Executive Summary

Between 2022-05-12 19:37 and 2022-05-13 12:06 UTC (a period of 16h 29min) notifications for feed updates were not delivered to most customers. These notifications were delivered between 12:16 and 12:36. No notifications were missed.

Technical Details

There was an attempt to enable DNSSEC for MX record lookups. However the applied configuration resulted in errors for domains without DNSSEC configured. This error was not caught during testing because it was expected that this configuration only validated for domains where it was configured and the haphazardly selected test set happened to only include validated domains.

This error was not noticed until 2022-05-13 12:00. This delay was due to quota exhaustion on FeedMail's primary error monitoring service. This resulted in errors not being reported immediately. Instead they were only detected by manual polling of our error dashboard.

Once the cause was identified the following actions were taken:

  1. All items in the unsent email queue were updated to ensure that they would not fail permanently.
  2. A version of FeedMail with DNSSEC disabled was deployed.
  3. All items in the unsent email queue were manually updated to send over the next 20 minutes.
  4. Logs were checked to identify if any entries were permanently dropped.

Action Items

Error Monitoring Quota Alerts

Right now we lack a good way to identify if we are approaching our primary error alerting quota at a rate that will result in quota depletion before the end of the month.

In this case a dependency update caused a high-frequency warning message which depleted our error quota over a few days early in the month. By the time it is clear that it was necessary to revert that dependency it was too late. Early warning of quota usage would have sufficiently mitigated this problem.

Backup Error Monitoring

Instead of relying on manual polling for backup monitoring an alert could have been set up in our logging framework to notify of this problem rapidly. This likely would have resulted in the error being fixed in 1 hour instead of 16.

This alert has now been set up.

Maintain Test MX List

This was also a preventable error. It was an intentional change to DNSSEC. However it happens that the handful of domains this was tested on all happened to have DNSSEC configured. In order to ensure sufficient testing in the future a list of "Text MXes" will be documented which contains accounts on publicly accessible MXes that exercise a variety of DNS configuration and mail software.

Re-Enable DNSSEC

DNSSEC is critical to the security of email. We will investigate how to restore DNSSEC validation to ensure the security of domains that support it. We are also considering alternate security mechanisms such as PGP encrypted notifications (let us know if you are interested)
As always, if you have any comments, questions or feedback simply reach out to FeedMail Support.
Labels:

Comments

Post a Comment

Popular posts from this blog

Delivery Delays to Gmail

In the past 48 hours Google has started delaying the delivery of some FeedMail notifications. This is currently affecting about 10% of messages to Gmail users. These notifications will be resent with a delay. We also speculate that some notifications will be marked as spam.   Update : As of 2023-05-09 this appears to be resolved. If You Are Affected If you use Gmail you may be affected by this. Notifications may be delayed or marked as spam. If your notifications are marked as spam you can create a filter to avoid this. Use "from:*@feedmail.org" as the rule and select " Never send it to Spam". If your notifications are delayed we are unaware of any action that you can take. However marking notifications that ended up in your spam folder as "Not Spam" may help avoid future delays.  It does appear that these emails are eventually being accepted but we are unsure if that means that they are actually ending up in users' mailboxes (or even their spam folder
Post a Comment
Read more

Updates to HTML Processing

Since its inception FeedMail has done processing on HTML content in feeds to ensure that it renders as expected in email form. At first this was fairly simple things like rewriting URLs to point to the correct location (many feeds use non-absolute URLs that won't work in email) but over time more complex transformations were added such as adding fallback content to media embeds without any. The full-text scraping feature requires even more complex processing as it requires stripping away most of the page and handling content that was designed for full-featured browsers. What changed? Recently FeedMail has migrated all HTML rewriting to use new infrastructure. This provides more flexibility and enabled new features (such as showing controls on all media embeds) and made our processing much more reliable. What does this mean to me? As a user you shouldn't see much difference. Overall the emails you receive should be better formatted but the difference will be subtle. Full-text sc
Post a Comment
Read more

Email Headers

FeedMail now sets some email headers that advanced users can use to filter the messages that FeedMail sends. For example you can filter notifications in a specific category into a different folder. The headers that we set are documented in the FAQ . These headers have been being set on notifications for a few months now so you can use past messages to test your filters. If you have any questions, or would like to see other headers set to help your filtering please let us know .
Post a Comment
Read more