Merge RSS feeds into one digest, deduplicated

If you follow more than a handful of feeds, you read the same story five times. Hacker News, Lobsters and two tech sites all link the same post within an hour, and a normal feed reader shows you every copy.

I wanted the opposite: one list, each story once, with every feed that carried it attached, and on the next run only what I have not seen yet. That is what my RSS Feed Aggregator actor on Apify does. Here is how to set it up from the "Daily news monitor" example task.

The input

The example task starts with two feeds and the "only new" memory switched on:

{
  "feedUrls": [
    "https://hnrss.org/frontpage",
    "https://feeds.arstechnica.com/arstechnica/index"
  ],
  "onlyNew": true,
  "stateKey": "news-monitor"
}

Replace the feeds with the ones you follow, up to 500 per run. RSS 2.0, Atom, RDF and JSON Feed all work, including Google News and Bing News search feeds. Give it feed URLs, not homepages: a homepage is reported as "not a feed" instead of being crawled.

stateKey is the name of this digest's memory. Use a different one for each topic or client, so a "competitors" digest and a "rust news" digest do not hide each other's stories.

What one merged story looks like

Here is a real row from a run on Hacker News and Lobsters, shortened:

{
  "rank": 1,
  "title": "Valen's Memory Safety: A New Kind of Borrow Checking",
  "url": "https://verdagon.dev/blog/valen-group-borrowing",
  "source": "Hacker News: Front Page",
  "publishedAt": "2026-10-11T12:06:22.000Z",
  "sourceCount": 2,
  "sources": [
    { "source": "Hacker News: Front Page", "publishedAt": "2026-10-11T12:06:22.000Z" },
    { "source": "Lobsters", "publishedAt": "2026-10-11T12:23:01.000Z" }
  ]
}

One row, two sources. The earliest copy becomes the main row, and stories carried by more feeds rank higher, then the newest first. sourceCount is a decent "how big is this" signal on its own.

How duplicates are found

Two items are treated as the same story when either of these is true:

1. Their links point to the same page after cleanup. Tracking parameters such as utm_*, fbclid and ref are removed, and so are AMP paths, www. and m. prefixes, trailing slashes and fragments.
2. Their headlines have the same words after cleanup. Case, accents, punctuation, word order, small words, "Updated:" and "Breaking:" prefixes and the publisher name at the end are ignored.

The rules are strict on purpose. Numbers must match, so "Rust 1.90" and "Rust 1.91" stay separate, and headlines more than 7 days apart are never merged by title. A wrong merge would hide a story from you; a missed merge only shows you a duplicate, so I chose the safer mistake.

Filter and schedule

Two optional fields narrow the digest:

{
  "keywords": ["rust", "open source"],
  "excludeKeywords": ["sponsored"],
  "maxAgeHours": 24
}

Keywords match whole words only, so rust does not match "trust", in any language and without caring about case or accents. They are searched in the title, summary and categories.

Then save the input as a task and add it to a schedule in Apify Console, daily or hourly. With onlyNew on and the same stateKey, each run returns only stories you have not received yet, including a late copy of an old story that shows up in another feed. Stories are remembered for 30 days.

To get the digest somewhere useful, set webhookUrl to a Zapier, Make or n8n catch hook, or your own endpoint. It is called only when a run has at least one new story, with up to 100 items in the body.

Broken feeds do not break the run

Feeds die all the time. When one does, the run continues and adds a free row that says why:

{ "recordType": "feed-error", "feedUrl": "https://example.com/", "error": "Not an RSS, Atom, RDF or JSON feed: the URL returned an HTML page" }

What it costs

Pricing is per event: $0.005 per run, $0.001 per feed read successfully, and $0.001 per story in the final digest after merging, filters and "only new". Broken feeds and duplicate rows are free. A one-off digest of 20 feeds with about 340 stories after merging costs about $0.37. The same 20 feeds every hour with "only new" and about 5 new stories per run cost about $0.03 per run.

Limits

It reads what the feed contains and never visits article pages, so there is no full article text unless the feed itself includes it (includeContent adds that). Up to 500 feeds per run; for more, split them over several tasks.

The actor is on Apify, and its page here shows real output and every release. If two stories got merged that should not have, or the other way around, send me both links at info@pankovs.com.