Skip to main content

Faster Updates

FeedMail has never had a user-configurable update rate option. This is because it never made sense to us. Why would you want anything but the fastest option? Update rate also isn't really a user-facing concern, it is more of an implementation detail. Users don't care how polling works, they just want to get their news. By keeping this decision internal it gives us the flexibility to introduce features such as instant updates via WebSub without requiring users to manually update their feed configuration.

We've recently made another improvement by reducing our poll rate limit from 15min to 5min. No action required on your part, your feeds are already updating faster. This means that you will get your updates three times faster for feeds that allow it. We have also adjusted the poll rate we use when feeds don't specify anything from 1h to as low as 5min for fast feeds.

The effect of these changes is significant. Excluding WebSub feeds (which are always updated in real-time). Over ¼ of feeds are now updated updated every 5min, ½ in 15min or less and ⅗ are updated at least every hour. Additionally if you focus on "Active" feeds (at least one post in the last 3 months) ½ are updated every 5min and ⅘ of active feeds are updated every 15min or less.

Chart of update rate of feeds.
The y-axis is the percentage of feeds that are updated at least that often.

Current Algorithm

For those of you who what the nitty-gritty details here is a snapshot of our algorithm. Note that we are constantly tweaking our heuristics to get the fastest updates to our customers without putting unnecessary loads on the sites that you are following. You can always find an up-to-date summary of our poll rate in our FAQ.

  1. If the feed supports WebSub.
    • WebSub will be used for real-time updates.
    • A backup poll will be performed weekly in case notifications are not sent.
    • Skip the rest of the algorithm.
  2. If a Cache-Control header with a max-age or an Expires header is present that will be used as the next poll time.
    • If slower than 24 hours it is set to 24 hours.
  3. Otherwise a default poll interval is used.
    • 5 minutes if both:
      • The feed responds in less than 1 second.
      • The response contains either an ETag or Last-Modified header.
    • 15 minutes if either of the above are true.
    • Otherwise 1 hour.
  4. The poll rate is then limited to:
    • 5 minutes if the feed responds in less than 1 second and has either an ETag or Last-Modified header.
    • Otherwise 15 minutes.
  5. Inactive feeds have their poll rate limited to 12h per year of inactivity. This slowdown is applied continuously, starting after a month of inactivity.

For Publishers

If you want to ensure that your viewers see your content ASAP we have the following recommendations.

  1. Support WebSub. This alone is sufficient for real-time updates not matter how fast your feed is, how often your post or any other factors. Additionally it reduces load on your server.
  2. Set a Cache-Control header with a max-age parameter specifying your desired update frequency. You can specify any value you want but some common values are shown below.
    • Every minute: Cache-Control: max-age=60
    • Every 15min: Cache-Control: max-age=900
    • Every 1h: Cache-Control: max-age=3600
  3. Support conditional requests via either ETag and If-None-Match or Last-Modified and If-Modified-Since.
  4. Ensure you feed is reliable. Feed readers and search engines will back off quickly on errors. Even a single error can result in multiple hours of back off, a couple hours of errors can result in days of delay before they notice that your feed is healthy again.
  5. Keep your feed consistently fast. Feed readers and search engines use response time and increases in response time to identify sites that are overloaded. By ensuring that your feed consistently loads in less that 1 second consumers will pick up your content as quickly as possible.


Popular posts from this blog

Delivery Delays to Gmail

In the past 48 hours Google has started delaying the delivery of some FeedMail notifications. This is currently affecting about 10% of messages to Gmail users. These notifications will be resent with a delay. We also speculate that some notifications will be marked as spam.   Update : As of 2023-05-09 this appears to be resolved. If You Are Affected If you use Gmail you may be affected by this. Notifications may be delayed or marked as spam. If your notifications are marked as spam you can create a filter to avoid this. Use "from:*" as the rule and select " Never send it to Spam". If your notifications are delayed we are unaware of any action that you can take. However marking notifications that ended up in your spam folder as "Not Spam" may help avoid future delays.  It does appear that these emails are eventually being accepted but we are unsure if that means that they are actually ending up in users' mailboxes (or even their spam folder

Updates to HTML Processing

Since its inception FeedMail has done processing on HTML content in feeds to ensure that it renders as expected in email form. At first this was fairly simple things like rewriting URLs to point to the correct location (many feeds use non-absolute URLs that won't work in email) but over time more complex transformations were added such as adding fallback content to media embeds without any. The full-text scraping feature requires even more complex processing as it requires stripping away most of the page and handling content that was designed for full-featured browsers. What changed? Recently FeedMail has migrated all HTML rewriting to use new infrastructure. This provides more flexibility and enabled new features (such as showing controls on all media embeds) and made our processing much more reliable. What does this mean to me? As a user you shouldn't see much difference. Overall the emails you receive should be better formatted but the difference will be subtle. Full-text sc

Email Headers

FeedMail now sets some email headers that advanced users can use to filter the messages that FeedMail sends. For example you can filter notifications in a specific category into a different folder. The headers that we set are documented in the FAQ . These headers have been being set on notifications for a few months now so you can use past messages to test your filters. If you have any questions, or would like to see other headers set to help your filtering please let us know .