Skip to main content

Delay on YouTube Feeds

Most YouTube feeds have not sent new notifications since 2023-10-02 14:20 UTC.

We will be triggering a manual YouTube feeds over the next hour and all missing notifications will be sent. If you have any missing notifications after 2023-10-05 14:00 UTC please reach out to support.

Update 14:00: All updates have been sent. If you believe that you are still missing updates feel free to reach out.

The rest of this post is a technical analysis of the issue.

Background

This was caused due to recent emergency response to YouTube WebSub notifications. The emergency response was necessary is due to the following factors.

  • YouTube WebSub posts are not spec compliant and do not contain the required information to send notifications. Therefore FeedMail uses these notifications as a "ping" to re-fetch the feed.
  • YouTube often sends notifications before the entries appear in the feed. The exact reason is not known but sometimes entries do not appear for up to an hour after the WebSub push.
  • YouTube also sends notifications for older items that are no longer in the feed.

In order to provide real-time updates for YouTube and to reduce resource usage, FeedMail works around these quirks. The workaround looks roughly like this:

  • When receiving a WebSub notification FeedMail re-checks the feed.
  • If the items mentioned in the notification are not present in the feed we poll with backoff, waiting for the item to appear.
  • FeedMail holds the HTTP connection open while doing this polling to avoid acknowledging a notification where we haven't seen the feed item. This way if the update fails the WebSub hub will notify us again and we can recheck.

Incident

Starting on September 27th YouTube started sending us an increasing volume of WebSub notifications.

Analysis showed that most of these notifications were for old videos. (Often years old.) This resulted in the full backoff loop running without finding the video (as it would never appear). It appears that many of these updates are retries of the same notification, likely because the long polling backoff resulted in a timeout in the WebSub hub. While in general the retries were useful for videos that weren't found they were undesired and high volume for old videos that would never appear on the feed.

Mitigation

In order to cut back this unnecessary work a code change was deployed that changed how we handled these "ping" notifications. (Which today only affects YouTube.)

  • Entries published more than 14 days ago are ignored.
  • Instead of backing off with retries an error is returned if the feed isn't found. This relies on the hub's built-in retry capabilities and avoids long-standing HTTP requests.
  • Entries updated more than 4 hours ago return a success, even if the entry wasn't found. This provides an upper-bound to retries.

Outage

Unfortunately there was a bug in the logic that filtered out entries published more than 14 days ago that caused nearly all WebSub notifications to be ignored.

YouTube feeds would still update whenever the WebSub subscription expires (every 7 days) but notifications wouldn't be sent until then.

Resolution

A new version was deployed that corrected the filter condition. After this point notifications started being received normally.

Additionally all YouTube feeds were scheduled to be re-checked over the next hour to catch any missed entries.
Labels:

Comments

Post a Comment

Popular posts from this blog

DNS Outage

From 2024-08-26 19:46 to 2024-08-27 11:21 UTC FeedMail had an outage. Until 2024-06-26 20:34 FeedMail was completely down. For the remainder of the outage most emails not sent. It is expected that no feed updates were lost during this outage. Updates would only be lost if they were only present on the feed within the 50min of total outage. Most feeds ensure that updates are present for days so this would not be an issue. Notifications have been delayed and should be sent by 2024-08-27 12:31. This may take longer if your mail provider applies limits and FeedMail needs to retry delivery at a later time. Update : All delayed notifications have been sent successfully. Timeline All times are in UTC . 2024-08-26 19:46 Start FeedMail goes down.   19:53 Detection Automated monitoring reported that feeds were not being checked. 20:34 The Database IP was hardcoded, restoring most functionality. 2024-08-27 11:21 Resolution FeedMail was switched external DNS. 11:24 Schedule of ...
Post a Comment
Read more

Digests Now Respect Category Filters

Due to an oversight category filters did not apply to digests. This has been corrected and future digests will be filtered by your selected categories. If you do not want this filtering to occur please update your filters to "Ignore selected categories" and deselect all categories to inactivate the filter.
Post a Comment
Read more

Digests are now Supported for Owner-Paid Feeds

Owner-paid feeds allow feed publishers to provide FeedMail to their subscribers at no cost. For example the FeedMail Blog is an owner-paid feed. Up until now digest subscriptions were not covered by owner-paid plans. Subscribers could select a digest but they would have to pay for the subscriptions themselves. Digests are now fully supported under owner-paid plans. For users: The owner-paid feeds in your digests no longer count towards the cost of the digest. For publishers: Users will now be able to receive your feed as a digest or included in one of their existing digests. You will be charged one credit for each digest issue containing items from your feed (no matter how many items from your feed are in that issue). Notably this cost will never be more than real-time subscriptions would be.
Post a Comment
Read more