Skip to main content

Notification Delivery Delay

Executive Summary

Between 2022-05-12 19:37 and 2022-05-13 12:06 UTC (a period of 16h 29min) notifications for feed updates were not delivered to most customers. These notifications were delivered between 12:16 and 12:36. No notifications were missed.

Technical Details

There was an attempt to enable DNSSEC for MX record lookups. However the applied configuration resulted in errors for domains without DNSSEC configured. This error was not caught during testing because it was expected that this configuration only validated for domains where it was configured and the haphazardly selected test set happened to only include validated domains.

This error was not noticed until 2022-05-13 12:00. This delay was due to quota exhaustion on FeedMail's primary error monitoring service. This resulted in errors not being reported immediately. Instead they were only detected by manual polling of our error dashboard.

Once the cause was identified the following actions were taken:

  1. All items in the unsent email queue were updated to ensure that they would not fail permanently.
  2. A version of FeedMail with DNSSEC disabled was deployed.
  3. All items in the unsent email queue were manually updated to send over the next 20 minutes.
  4. Logs were checked to identify if any entries were permanently dropped.

Action Items

Error Monitoring Quota Alerts

Right now we lack a good way to identify if we are approaching our primary error alerting quota at a rate that will result in quota depletion before the end of the month.

In this case a dependency update caused a high-frequency warning message which depleted our error quota over a few days early in the month. By the time it is clear that it was necessary to revert that dependency it was too late. Early warning of quota usage would have sufficiently mitigated this problem.

Backup Error Monitoring

Instead of relying on manual polling for backup monitoring an alert could have been set up in our logging framework to notify of this problem rapidly. This likely would have resulted in the error being fixed in 1 hour instead of 16.

This alert has now been set up.

Maintain Test MX List

This was also a preventable error. It was an intentional change to DNSSEC. However it happens that the handful of domains this was tested on all happened to have DNSSEC configured. In order to ensure sufficient testing in the future a list of "Text MXes" will be documented which contains accounts on publicly accessible MXes that exercise a variety of DNS configuration and mail software.

Re-Enable DNSSEC

DNSSEC is critical to the security of email. We will investigate how to restore DNSSEC validation to ensure the security of domains that support it. We are also considering alternate security mechanisms such as PGP encrypted notifications (let us know if you are interested)
As always, if you have any comments, questions or feedback simply reach out to FeedMail Support.
Labels:

Comments

Post a Comment

Popular posts from this blog

DNS Outage

From 2024-08-26 19:46 to 2024-08-27 11:21 UTC FeedMail had an outage. Until 2024-06-26 20:34 FeedMail was completely down. For the remainder of the outage most emails not sent. It is expected that no feed updates were lost during this outage. Updates would only be lost if they were only present on the feed within the 50min of total outage. Most feeds ensure that updates are present for days so this would not be an issue. Notifications have been delayed and should be sent by 2024-08-27 12:31. This may take longer if your mail provider applies limits and FeedMail needs to retry delivery at a later time. Update : All delayed notifications have been sent successfully. Timeline All times are in UTC . 2024-08-26 19:46 Start FeedMail goes down.   19:53 Detection Automated monitoring reported that feeds were not being checked. 20:34 The Database IP was hardcoded, restoring most functionality. 2024-08-27 11:21 Resolution FeedMail was switched external DNS. 11:24 Schedule of ...
Post a Comment
Read more

Digests Now Respect Category Filters

Due to an oversight category filters did not apply to digests. This has been corrected and future digests will be filtered by your selected categories. If you do not want this filtering to occur please update your filters to "Ignore selected categories" and deselect all categories to inactivate the filter.
Post a Comment
Read more

Digests are now Supported for Owner-Paid Feeds

Owner-paid feeds allow feed publishers to provide FeedMail to their subscribers at no cost. For example the FeedMail Blog is an owner-paid feed. Up until now digest subscriptions were not covered by owner-paid plans. Subscribers could select a digest but they would have to pay for the subscriptions themselves. Digests are now fully supported under owner-paid plans. For users: The owner-paid feeds in your digests no longer count towards the cost of the digest. For publishers: Users will now be able to receive your feed as a digest or included in one of their existing digests. You will be charged one credit for each digest issue containing items from your feed (no matter how many items from your feed are in that issue). Notably this cost will never be more than real-time subscriptions would be.
Post a Comment
Read more