Skip to main content

Faster Updates

FeedMail has never had a user-configurable update rate option. This is because it never made sense to us. Why would you want anything but the fastest option? Update rate also isn't really a user-facing concern, it is more of an implementation detail. Users don't care how polling works, they just want to get their news. By keeping this decision internal it gives us the flexibility to introduce features such as instant updates via WebSub without requiring users to manually update their feed configuration.

We've recently made another improvement by reducing our poll rate limit from 15min to 5min. No action required on your part, your feeds are already updating faster. This means that you will get your updates three times faster for feeds that allow it. We have also adjusted the poll rate we use when feeds don't specify anything from 1h to as low as 5min for fast feeds.

The effect of these changes is significant. Excluding WebSub feeds (which are always updated in real-time). Over ¼ of feeds are now updated updated every 5min, ½ in 15min or less and ⅗ are updated at least every hour. Additionally if you focus on "Active" feeds (at least one post in the last 3 months) ½ are updated every 5min and ⅘ of active feeds are updated every 15min or less.

Chart of update rate of feeds.
The y-axis is the percentage of feeds that are updated at least that often.

Current Algorithm

For those of you who what the nitty-gritty details here is a snapshot of our algorithm. Note that we are constantly tweaking our heuristics to get the fastest updates to our customers without putting unnecessary loads on the sites that you are following. You can always find an up-to-date summary of our poll rate in our FAQ.

  1. If the feed supports WebSub.
    • WebSub will be used for real-time updates.
    • A backup poll will be performed weekly in case notifications are not sent.
    • Skip the rest of the algorithm.
  2. If a Cache-Control header with a max-age or an Expires header is present that will be used as the next poll time.
    • If slower than 24 hours it is set to 24 hours.
  3. Otherwise a default poll interval is used.
    • 5 minutes if both:
      • The feed responds in less than 1 second.
      • The response contains either an ETag or Last-Modified header.
    • 15 minutes if either of the above are true.
    • Otherwise 1 hour.
  4. The poll rate is then limited to:
    • 5 minutes if the feed responds in less than 1 second and has either an ETag or Last-Modified header.
    • Otherwise 15 minutes.
  5. Inactive feeds have their poll rate limited to 12h per year of inactivity. This slowdown is applied continuously, starting after a month of inactivity.

For Publishers

If you want to ensure that your viewers see your content ASAP we have the following recommendations.

  1. Support WebSub. This alone is sufficient for real-time updates not matter how fast your feed is, how often your post or any other factors. Additionally it reduces load on your server.
  2. Set a Cache-Control header with a max-age parameter specifying your desired update frequency. You can specify any value you want but some common values are shown below.
    • Every minute: Cache-Control: max-age=60
    • Every 15min: Cache-Control: max-age=900
    • Every 1h: Cache-Control: max-age=3600
  3. Support conditional requests via either ETag and If-None-Match or Last-Modified and If-Modified-Since.
  4. Ensure you feed is reliable. Feed readers and search engines will back off quickly on errors. Even a single error can result in multiple hours of back off, a couple hours of errors can result in days of delay before they notice that your feed is healthy again.
  5. Keep your feed consistently fast. Feed readers and search engines use response time and increases in response time to identify sites that are overloaded. By ensuring that your feed consistently loads in less that 1 second consumers will pick up your content as quickly as possible.

Comments

Popular posts from this blog

Digests are Coming

Up to this point FeedMail has only supported real-time notifications. Meaning that every feed update immediately produces a single email. However this is about to change! When we asked for feedback on the features you would like to see in FeedMail we had a number of users reach out saying that they wanted a way to batch notifications together. We saw two main reasons for this: To reduce noise in their inbox. For some high-volume feeds users wanted to be able to quickly skim, then delete the entire batch in one go. While deleting one-by-one offers more flexibility, the bulk option is easier for high-volume feeds. To reduce costs. While we believe that our prices are incredibly reasonable, they can add up if you are getting lots of updates. For example if you follow a feed that updates every 15min that will be about $35 a year (or half price if you buy your credits in bulk). Not super expensive but maybe more than you want to spend for a single feed! Digests provide and option for cost

Digests Leave Beta

Thanks everyone who has helped evaluate digests over the past weeks. All of the blocking issues are now resolved and we will be releasing them soon. Once digests are officially released there will be links to them from the FeedMail site and pricing information added to our homepage. Price Increase Part of the purpose of the beta was to evaluate the cost of providing digests and see how they would be used. We have decided upon final pricing which we hope will be sustainable for years to come. Digests issues will cost 1 credit per 5 feeds. Note that this is feeds included in an issue , not total feeds that target a particular digest. It also does not matter how many new items a feed has. So if you have a digest with 200 feeds configured but this morning's issue only has new items from 2 of them it will cost 1 credit. If 14 feeds update the next day that issue will cost 3 credits. If the day after has no updates it will cost nothing. This new pricing takes effect no earlier than 202

Update to Date-based Entry Ignoring

TL;DR FeedMail will now ignore new items 7 days older than a previously seen item. This is expected to affect almost no "true" new posts. In theory checking to find new entries for a feed is a simple process. Download the feed. Check the ID of each entry to see if you have seen it before. However the real world is much messier. It is recommended for feed IDs to be URLs (to ensure global uniqueness) however this results in many feeds just using the URL that the article is available at. However these URLs sometimes change, and poorly designed feed generators update the ID of existing entries to the new URL. From a protocol point of view these are completely new entries, however to a user these are duplicates. In order to reduce the effect of this common issue on our our users FeedMail has some simple mitigations for posts that have recorded published dates. If the entry is older than a year always ignore it. If the entry is older than the 10th newest post in the feed ignore it.