Skip to main content

Update to Date-based Entry Ignoring

TL;DR FeedMail will now ignore new items 7 days older than a previously seen item. This is expected to affect almost no "true" new posts.

In theory checking to find new entries for a feed is a simple process.

  1. Download the feed.
  2. Check the ID of each entry to see if you have seen it before.

However the real world is much messier. It is recommended for feed IDs to be URLs (to ensure global uniqueness) however this results in many feeds just using the URL that the article is available at. However these URLs sometimes change, and poorly designed feed generators update the ID of existing entries to the new URL.

From a protocol point of view these are completely new entries, however to a user these are duplicates. In order to reduce the effect of this common issue on our our users FeedMail has some simple mitigations for posts that have recorded published dates.

  1. If the entry is older than a year always ignore it.
  2. If the entry is older than the 10th newest post in the feed ignore it.
  3. If the entry is more than 7 days older than the newest post in the feed ignore it.

Rule number 3 is a new rule to better filter out old entries for infrequent feeds. In many cases this will ensure that you only receive one or two duplicate entries if the feed accidentally changes their post IDs.

One might think that you can simply ignore any article that isn't newer than the newest article, however in practice many feeds post items with the published timestamp slightly out of order. Based on our analysis a 7 day window will almost never ignore new articles while filtering out a lot of duplicates with new IDs.

Labels:

Comments

Post a Comment

Popular posts from this blog

Delay on YouTube Feeds

Most YouTube feeds have not sent new notifications since 2023-10-02 14:20 UTC. We will be triggering a manual YouTube feeds over the next hour and all missing notifications will be sent. If you have any missing notifications after 2023-10-05 14:00 UTC please reach out to support. Update 14:00 : All updates have been sent. If you believe that you are still missing updates feel free to reach out. The rest of this post is a technical analysis of the issue. Background This was caused due to recent emergency response to YouTube WebSub notifications. The emergency response was necessary is due to the following factors. YouTube WebSub posts are not spec compliant and do not contain the required information to send notifications. Therefore FeedMail uses these notifications as a "ping" to re-fetch the feed. YouTube often sends notifications before the entries appear in the feed. The exact reason is not known but sometimes entries do not appear for up to an hour after the WebSub push. ...
Post a Comment
Read more

Invalid DKIM Signature for Some Mail

As of 2023-07-26 FeedMail messages sent via AWS SES have an invalid feedmail.org DKIM signature. This may result in messages ending up in your spam folder. This is an ongoing issue and updates will be posted here as they are available. 2023-09-25 We have issue an update that should avoid SES re-formatting signed fields. This is a workaround but should result in valid signatures. Who is affected? This is unlikely to affect most of our users for the following reasons: 97% of our mail is sent by us directly, not via AWS. This mail still has a SPF-verified sending IP. However this may still affect users because spam filters may consider these messages less "good" than they would have been with a valid DKIM signature. The most notable Inbox Providers affected by this are Apple and Microsoft. FeedMail uses AWS SES for these providers as they reject all messages from our network provider. However other smaller providers may also have a portion of their messages sent via AWS SES. ...
Post a Comment
Read more

Support for SMTP MTA Strict Transport Security

FeedMail now supports SMTP MTA Strict Transport Security (MTA-STS) . This standard provides receiving domains a way to attempt to indicate that incoming mail should only be delivered over a secure connection. When FeedMail receives this signal it will refuse to deliver over insecure connections (retrying mail as required). FeedMail does not currently support SMTP TLS Reporting (TLSRPT) . We will be reaching out to any existing customers who's mail may be rejected due to this change.
Post a Comment
Read more