How to fix broken internal links at scale

Group by cause, not by instance

The first move on a crawl report with thousands of broken links is to collapse it. Count the distinct broken destinations, not the total number of links.

A report showing 4,000 broken links commonly resolves to fewer than twenty unique destinations, each linked from hundreds of pages by a shared template. Twenty fixes, not four thousand.

Group the report by destination URL, then look at what the linking pages have in common. If they share a template, the fix is in the template.

The usual structural causes

A navigation or footer link to a removed page. Appears on every page on the site instantly.

A related-content block generating links from stale data — a cached list, a search index that has not been rebuilt, a database table not cleaned up after deletions.

Category or filter links generated for combinations that produce no results and return errors rather than empty states.

Pagination generating links beyond the last real page.

Hard-coded links in a template or component, often pointing at a URL structure retired in a previous rebuild.

Fix in the right layer

Once grouped, each cause has a layer:

• Navigation and footer — fix the menu definition, once
• Generated blocks — fix the query or rebuild the index; do not patch the output
• Filter and pagination — make the generator only emit links to combinations that exist
• Hard-coded — search the codebase for the URL and correct every occurrence

Adding redirects for generated broken links is treating a symptom. The generator will keep producing them.

Verify with a diff, not a spot check

On a large site, spot-checking a few pages after a fix proves nothing. Crawl before and after, and diff the two reports:

sort before.csv > b.txt; sort after.csv > a.txt
comm -23 b.txt a.txt | wc -l   # fixed
comm -13 b.txt a.txt | wc -l   # newly broken

The second number matters as much as the first. Template fixes routinely break something else, and a diff is the only reliable way to see it.

Then instrument it

At this scale, manual monthly crawls are not enough. Practical measures:

• Log 404s with their referrer and alert when the rate rises
• Run a crawl on a schedule and compare against the previous run automatically
• Add a build-time check that fails if a template references a URL that does not resolve

That last one prevents the whole class of problem, which is worth more than any amount of fixing.

What this costs

Work at this scale sits in Authority Builder — $12,500 setup, $3,500 a month, where the technical layer is maintained rather than repaired periodically. Growth Engine at $5,500 and $1,500 covers regular crawling and fixing for mid-sized sites.