Skip to main content

Faster Updates

FeedMail has never had a user-configurable update rate option. This is because it never made sense to us. Why would you want anything but the fastest option? Update rate also isn't really a user-facing concern, it is more of an implementation detail. Users don't care how polling works, they just want to get their news. By keeping this decision internal it gives us the flexibility to introduce features such as instant updates via WebSub without requiring users to manually update their feed configuration.

We've recently made another improvement by reducing our poll rate limit from 15min to 5min. No action required on your part, your feeds are already updating faster. This means that you will get your updates three times faster for feeds that allow it. We have also adjusted the poll rate we use when feeds don't specify anything from 1h to as low as 5min for fast feeds.

The effect of these changes is significant. Excluding WebSub feeds (which are always updated in real-time). Over ¼ of feeds are now updated updated every 5min, ½ in 15min or less and ⅗ are updated at least every hour. Additionally if you focus on "Active" feeds (at least one post in the last 3 months) ½ are updated every 5min and ⅘ of active feeds are updated every 15min or less.

Chart of update rate of feeds.
The y-axis is the percentage of feeds that are updated at least that often.

Current Algorithm

For those of you who what the nitty-gritty details here is a snapshot of our algorithm. Note that we are constantly tweaking our heuristics to get the fastest updates to our customers without putting unnecessary loads on the sites that you are following. You can always find an up-to-date summary of our poll rate in our FAQ.

  1. If the feed supports WebSub.
    • WebSub will be used for real-time updates.
    • A backup poll will be performed weekly in case notifications are not sent.
    • Skip the rest of the algorithm.
  2. If a Cache-Control header with a max-age or an Expires header is present that will be used as the next poll time.
    • If slower than 24 hours it is set to 24 hours.
  3. Otherwise a default poll interval is used.
    • 5 minutes if both:
      • The feed responds in less than 1 second.
      • The response contains either an ETag or Last-Modified header.
    • 15 minutes if either of the above are true.
    • Otherwise 1 hour.
  4. The poll rate is then limited to:
    • 5 minutes if the feed responds in less than 1 second and has either an ETag or Last-Modified header.
    • Otherwise 15 minutes.
  5. Inactive feeds have their poll rate limited to 12h per year of inactivity. This slowdown is applied continuously, starting after a month of inactivity.

For Publishers

If you want to ensure that your viewers see your content ASAP we have the following recommendations.

  1. Support WebSub. This alone is sufficient for real-time updates not matter how fast your feed is, how often your post or any other factors. Additionally it reduces load on your server.
  2. Set a Cache-Control header with a max-age parameter specifying your desired update frequency. You can specify any value you want but some common values are shown below.
    • Every minute: Cache-Control: max-age=60
    • Every 15min: Cache-Control: max-age=900
    • Every 1h: Cache-Control: max-age=3600
  3. Support conditional requests via either ETag and If-None-Match or Last-Modified and If-Modified-Since.
  4. Ensure you feed is reliable. Feed readers and search engines will back off quickly on errors. Even a single error can result in multiple hours of back off, a couple hours of errors can result in days of delay before they notice that your feed is healthy again.
  5. Keep your feed consistently fast. Feed readers and search engines use response time and increases in response time to identify sites that are overloaded. By ensuring that your feed consistently loads in less that 1 second consumers will pick up your content as quickly as possible.

Comments

Popular posts from this blog

Announcing FeedMail

I'm pleased to be sharing a project that I have been working on for a while and have been thinking about doing for even longer. FeedMail is a simple service that aims to get updates from your favourite websites to your email with no fuss and no nonsense. If you are already sold and want to follow some feeds simply go to feedmail.org to get started. How FeedMail Works FeedMail works using a set of technologies informally called RSS. FeedMail actually supports a variety of feed formats including Atom, RSS2 and RSS1. These feeds are created by websites and updated whenever new content is posted. FeedMail subscribes to these feeds on your behalf and forwards new entries to the email address of your choice. Many websites support these feeds. Just post the URL to an article or website that you want to subscribe to and FeedMail will show you the available feeds. For example the following websites support RSS: YouTube Channels Medium GitHub Releases Tumblr Many news sites Many more... RSS

Providing Email Subscriptions to your Readers with FeedMail

If you offer a blog with an RSS feed you can reach more users by offering email subscription as well. FeedMail provides an easy-to-integrate newsletter that has no cost to you. Just select one of the implementation strategies below. Implementation Options Link The easiest option is to provide a link to the subscribe page. First got to the FeedMail New Subscription page. Enter your website's URL and click "Go".  Then copy the URL from your browser's address bar. This the the URL to subscribe to your website! It should look something like https://feedmail.org/subscriptions/new?url= https%3A%2F%2Fyour-site.example . You can then link to that URL from your site. For example if you are using raw HTML in your design it would look something like: <a href="https://feedmail.org/subscriptions/new?url= https%3A%2F%2Fyour-site.example ">Click here to subscribe by email.</a>  Subscribe Button To use a button simply add the following code to your website. &l

How to Subscribe to YouTube via Email

YouTube disabled native email notifications in August 2020 . Luckily FeedMail lets you get these notifications back, and they are better than ever! Why Email Notifications? At FeedMail were are a huge fan of email notifications! Synced across all of your devices. Easy to sort and filter. Very customizable. You can decide if you want them to buzz your phone, ding your computer, both, or quietly file into a folder for you to check when you have some downtime. One place for all of your updates. No need to remember to log into YouTube, Odysee, Vimeo, PeerTube and Nebula on every device. Better than Native? We think so! No need to create a YouTube account. Per-video notifications, no batching to mess up your filters and workflow. You can pick a different email address for each channel for easy filtering. Not everything needs to go to your primary address. Instructions Find the channel that you want to subscribe to. If you are watching a video you can get to the channel but clicking the cha