How I catch sitemap regressions before Google Search Console does

Google Search Console's indexing report is often weeks behind. If a migration leaves a noindex tag on a page, or a redirect chain breaks, you usually find out from GSC long after it mattered, or from a support ticket.

I started tracking this differently. Instead of a one-off crawl, I run a scheduled scan of the sitemap and keep state between runs. Each run checks every URL for broken links, 4xx and 5xx errors, redirect chains, noindex tags, canonical mismatches, and robots.txt disallow conflicts, then diffs the result against the previous run.

The diff is the part that matters. A one-off audit gives you a wall of issues, most of which you already know about and have decided to live with. A diff only tells you what changed since last time: new breakage, issues that are still open, and issues that got fixed. That is a much smaller, much more actionable list, and it is small enough to wire into a webhook alert without drowning in noise.

A few things I learned building this.

Robots.txt disallow conflicts are undercounted as a bug class. A page can be perfectly indexable and still get silently dropped because a disallow rule added for a staging path also matches a production route.

Redirect chains tend to grow during migrations and nobody removes the old hops afterward. Three or four redirect hops on a page is common and nobody notices until it is flagged.

Treating every scan as a one-off wastes the signal you already paid for. The value of monitoring is almost entirely in the second run and every run after that, not the first one.

I packaged this as Sitemap Health Monitor, a pay-per-URL Apify actor with no subscription: $0.50 per 1,000 URLs checked and no start fee, so a daily scan of 500 pages costs about $0.25 a day, about $7.50 a month. If you are dealing with sitemap rot after a migration or want same-day signal instead of waiting on GSC, it is on Apify, and its page here shows a real run and every release.

It diffs hreflang pairs the same way: language versions that do not link back, return an error, redirect or are missing from the sitemap. Curious how other people handle this. Anything else worth diffing run over run beyond what is listed above? Write to info@pankovs.com.