How do you audit an XML sitemap?
Fetch it as a crawler does and confirm a 200 status with an XML content type and no redirect. Then status-test every URL: anything returning 301, 404 or 410, anything canonicalised elsewhere, anything noindexed and anything blocked in robots.txt does not belong. Finally, diff against a crawl to find what is missing.
Fetch it as a crawler would
Start with curl, not a browser, so you see the raw bytes and headers:
curl -sI https://example.com/sitemap.xml
curl -s https://example.com/sitemap.xml | head -50
Check: a 200 status, Content-Type of application/xml or text/xml, no redirect chain, and no login wall. A sitemap that redirects or returns 404 is invisible no matter how good its contents are.
Check the structural limits
- Maximum 50,000 URLs per file
- Maximum 50MB uncompressed
- All URLs in one file must share a protocol and host
- Index files may not nest more than one level deep
If you are over the limits, split by section and publish a sitemap index. Splitting by content type is more useful than splitting arbitrarily, because it lets you compare submitted against indexed per section in Search Console.
Test every URL in it
This is the audit. Extract the URLs and check the status of each:
curl -s https://example.com/sitemap.xml \
| grep -oP '(?<=<loc>)[^<]+' \
| while read u; do echo "$(curl -s -o /dev/null -w '%{http_code}' "$u") $u"; done
Everything should return 200. Anything returning 301, 302, 404 or 410 does not belong in a sitemap — a sitemap is a list of final destinations, not a redirect map.
Then check the harder failures
Status 200 is not sufficient. Also remove:
URLs canonicalised elsewhere. If a page's canonical tag points to a different URL, listing it contradicts your own signal.
Noindexed URLs. Telling a crawler to index a page you have told it not to index is a direct conflict.
Blocked URLs. Anything disallowed in robots.txt cannot be crawled, so listing it is noise.
Parameterised duplicates. Session IDs, tracking parameters, sort orders.
Thin or placeholder pages. Empty tag archives, paginated shells, staging leftovers.
Check what is missing
The reverse test is the one most audits skip. Crawl the site, list every indexable URL, and diff against the sitemap. Pages that should be indexed but are absent are usually orphans — no internal links either — which is a bigger problem than the sitemap.
Get lastmod right or drop it
lastmod is used as a hint about when to recrawl. If it updates to today's date on every build regardless of whether content changed, it is misinformation and will eventually be ignored. Either set it from real content-modification timestamps, or omit the element.
priority and changefreq are effectively ignored. Do not spend time on them.
Confirm it is discoverable
Reference it from robots.txt with an absolute URL, and submit it in Search Console. Then compare submitted against indexed — a large gap points at quality or crawl problems, not at the sitemap itself.
Frequently asked questions
What belongs in a sitemap?
URLs that return 200, are self-canonical, are not noindexed, are not blocked in robots.txt, and are pages you genuinely want indexed. Everything else is noise that dilutes the signal.
Should redirects be listed?
No. A sitemap is a list of final destinations. Anything returning 301, 302, 404 or 410 should be removed and replaced with its destination if that destination should be indexed.
What are the size limits?
50,000 URLs and 50MB uncompressed per file. Beyond that, split by section and publish a sitemap index — splitting by content type lets you compare submitted against indexed per section.
Does lastmod matter?
Only if it is accurate. A lastmod that changes on every build regardless of content is misinformation and will be discounted. Set it from real modification timestamps or omit the element entirely.
What if submitted and indexed counts differ a lot?
That usually points at page quality or crawl budget rather than the sitemap. The sitemap is a request to crawl, not an instruction to index.