Faster Updates

FeedMail has never had a user-configurable update rate option. This is because it never made sense to us. Why would you want anything but the fastest option? Update rate also isn't really a user-facing concern, it is more of an implementation detail. Users don't care how polling works, they just want to get their news. By keeping this decision internal it gives us the flexibility to introduce features such as instant updates via WebSub without requiring users to manually update their feed configuration.

We've recently made another improvement by reducing our poll rate limit from 15min to 5min. No action required on your part, your feeds are already updating faster. This means that you will get your updates three times faster for feeds that allow it. We have also adjusted the poll rate we use when feeds don't specify anything from 1h to as low as 5min for fast feeds.

The effect of these changes is significant. Excluding WebSub feeds (which are always updated in real-time). Over ¼ of feeds are now updated updated every 5min, ½ in 15min or less and ⅗ are updated at least every hour. Additionally if you focus on "Active" feeds (at least one post in the last 3 months) ½ are updated every 5min and ⅘ of active feeds are updated every 15min or less.

The y-axis is the percentage of feeds that are updated at least that often.

Current Algorithm

For those of you who what the nitty-gritty details here is a snapshot of our algorithm. Note that we are constantly tweaking our heuristics to get the fastest updates to our customers without putting unnecessary loads on the sites that you are following. You can always find an up-to-date summary of our poll rate in our FAQ.

1. If the feed supports WebSub.
• WebSub will be used for real-time updates.
• A backup poll will be performed weekly in case notifications are not sent.
• Skip the rest of the algorithm.
  
2. If a Cache-Control header with a max-age or an Expires header is present that will be used as the next poll time.
• If slower than 24 hours it is set to 24 hours.
3. Otherwise a default poll interval is used.
• 5 minutes if both:
• The feed responds in less than 1 second.
• The response contains either an ETag or Last-Modified header.
• 15 minutes if either of the above are true.
  
• Otherwise 1 hour.
4. The poll rate is then limited to:
• 5 minutes if the feed responds in less than 1 second and has either an ETag or Last-Modified header.
• Otherwise 15 minutes.
5. Inactive feeds have their poll rate limited to 12h per year of inactivity. This slowdown is applied continuously, starting after a month of inactivity.
   

For Publishers

If you want to ensure that your viewers see your content ASAP we have the following recommendations.

1. Support WebSub. This alone is sufficient for real-time updates not matter how fast your feed is, how often your post or any other factors. Additionally it reduces load on your server.
2. Set a Cache-Control header with a max-age parameter specifying your desired update frequency. You can specify any value you want but some common values are shown below.
   
• Every minute: Cache-Control: max-age=60
• Every 15min: Cache-Control: max-age=900
• Every 1h: Cache-Control: max-age=3600
3. Support conditional requests via either ETag and If-None-Match or Last-Modified and If-Modified-Since.
   
4. Ensure you feed is reliable. Feed readers and search engines will back off quickly on errors. Even a single error can result in multiple hours of back off, a couple hours of errors can result in days of delay before they notice that your feed is healthy again.
5. Keep your feed consistently fast. Feed readers and search engines use response time and increases in response time to identify sites that are overloaded. By ensuring that your feed consistently loads in less that 1 second consumers will pick up your content as quickly as possible.