What Distinguishes Broken Links, 404 and Soft 404
Before monitoring anything, a clean distinction helps. A 404 is the standard HTTP response a server returns when a requested resource does not exist - the status code tells browser and search engine unambiguously: this page is gone. A broken link (dead link) is a reference - internal or external - that points to such an unreachable target: a 404, a 410 (permanently removed), a server error (5xx) or an unresolvable hostname. The two terms overlap but describe different perspectives: the 404 is the response, the broken link is the reference to it.
More insidious is the soft 404. Here the server returns a success status code 200 even though the page is effectively empty or shows an error message such as 'page not found'. To the visitor it looks like an error, to the search engine like a valid page - and this very discrepancy is what makes soft 404s so problematic. Google has to load, render, evaluate and then discard such pages, consuming by its own account a multiple of the crawl resources of a clean 404 response (Google Search Central). A clear 404 status code, by contrast, is a strong signal for Google to crawl that URL less often in future (Google Search Central).
| Type | What the server returns | Problem |
|---|---|---|
| Hard 404 | Status code 404 / 410 | Reference into the void, but honestly signalled |
| Soft 404 | Status code 200 + error content | Search engine treats error page as valid |
| Dead external link | Timeout / DNS error / 5xx | Third-party page gone, hard to find manually |
| Redirect chain to 404 | 301/302 -> ... -> 404 | Link equity dissipates, target no longer exists |
| Broken anchor / asset | 404 on image, CSS, script | Layout breaks, page looks broken |
This distinction is more than terminology. Anyone who only watches for visible 404 pages misses the soft 404s and redirect chains hiding inside seemingly working 200 responses. Robust monitoring has to cover all variants - not just the obvious case.
Why Dead Links Cost Ranking and Crawl Budget
Search engines follow links to discover and evaluate content. When the crawler hits a dead link, it wastes part of the crawl budget reserved for your domain - the number of URLs a search engine fetches within a time window (Google Search Central). Every 404 URL, every redirect chain and every orphaned page eats into resources that were meant to index your important content. For large shops and portals with tens of thousands of URLs this quickly matters: if Google crawls in circles through dead references, new or updated pages stay unindexed for longer.
The prevalence of the problem is well documented. An analysis by Semrush found that around 35% of crawled websites had broken internal links with bad HTTP status codes, and roughly 70% of those returned a 4xx code (Semrush). An Ahrefs study on link ageing concluded that at least 66.5% of links to websites over the past roughly nine years are dead or broken (Ahrefs link rot study). And the 38% of pages that vanished within a decade (Pew Research 2024) show that external references inevitably rot over time - an effect often called 'link rot'.
Google makes clear that a 404 in itself is not a quality or ranking signal - every website has dead pages, and that is to be expected (Google Search Central / John Mueller). The damage is indirect: soft 404s keep being crawled and consume crawl budget, while a clean 404 is a strong signal to fetch that URL less often in future (Google Search Central). Serving too many error URLs therefore risks not a direct penalty but that important pages get crawled and indexed less often and later - on top of the tangible UX and conversion losses.
On top comes an indirect effect that is often underestimated: backlinks pointing from outside to a URL that has since vanished no longer pass on their link equity meaningfully once the target runs into a 404. Laboriously earned references from other websites quietly fizzle out. A well-maintained set of redirects captures this link equity and routes it to the appropriate new page - provided you notice the dead reference at all. This is exactly where continuous monitoring helps: it reveals which previously linked targets have broken away, so a redirect can be added in a targeted way instead of letting the earned authority decay.
The UX Cost: Lost Trust and Abandoned Conversions
Behind every dead link is a person who wanted to achieve something and instead lands on an error page. The consequences for user experience are considerable: around 88% of users do not return to a website after a poor online experience (Amazon Web Services / Forrester). A 404 page in the middle of a purchase or after clicking a search ad is exactly such an experience - the visitor is lost, often for good.
It gets particularly expensive in e-commerce. Technical hurdles in the ordering process typically lead straight to abandonment, and the average cart abandonment rate across industries is around 70% (Baymard Institute). A dead link on a product, category or checkout page adds directly to this already high abandonment rate. Anyone investing in paid campaigns pays twice: first for the click, then for the lost customer who lands on a 404.
A dead link in a running search ad, a social ad or a sent newsletter burns budget in real time - every click costs money and leads nowhere. Unlike organic traffic, the advertising euro already spent cannot be recovered. Campaign landing pages in particular should therefore be monitored closely.
On top comes the loss of trust: a broken page reads as a sign of carelessness. Visitors quickly transfer the impression of a broken website onto the reliability of the whole offering - an effect that is hard to quantify but weighs heavily in the competition for credibility. How closely technical reliability and perception are linked is explored further in the article on uptime monitoring and why it matters.
Why Deploys and Migrations Create Dead Links
Dead links rarely stem from negligence - usually they are a side effect of normal work on a website. Any change to structure, content or technology can break references that worked before. The following patterns appear most often in practice.
A relaunch or CMS switch changes the URL structure. Without complete redirects, old internal links, bookmarks and backlinks suddenly point to 404.
A product leaves the range, a blog post is deleted - but references from navigation, footer or other articles remain and turn dead.
A linked third-party page moves or goes offline. Without an automatic check, such external dead links typically go unnoticed - until a visitor reports them.
A wrong path in the template, a lost asset after the build or a mistyped anchor produces 404 on scripts, images or subpages.
The release frequency of modern teams makes this worse. According to the DORA report, high-performing teams deploy multiple times a day (DORA State of DevOps 2024). Each of those deploys is an opportunity to silently move a URL or break a reference. The pattern resembles other creeping configuration deviations after releases - for example consent and tag drift after deploy, where a harmless change likewise has unintended consequences.
Why Manual Checking Is Not Enough
The obvious answer - 'we'll just click through the important pages' - does not scale. Even a mid-sized website quickly has several thousand internal and external references. Nobody checks those by hand after every deploy. And even a thorough visual check only captures the state at one point in time: the external link that works today can be dead tomorrow without anything changing on your own site.
- Scale - thousands of links cannot be checked regularly by hand, least of all after every release
- External references - dead third-party pages arise without your involvement and typically stay invisible in a pure code review
- Soft 404 - return status code 200 and do not stand out at all in a superficial check
- JavaScript navigation - links created only by a script are missed by plain command-line HTML fetches
- Time factor - a dead link noticed only after weeks has by then cost ranking and conversions
The last point is decisive: the time window. Without monitoring, a dead link is often only noticed when a customer complains or the ranking has already dropped. By then visitors have seen the error page, search engines have wasted crawl budget, and campaign clicks have led nowhere. Continuous monitoring shortens this window from weeks to a few hours.
Detecting Dead Links Reliably in Technical Terms
Robust detection starts with crawling the website systematically: starting from the homepage, all reachable internal links are followed, collected and individually checked for their status code. External references are additionally requested to capture timeouts, DNS errors and 4xx/5xx responses from third parties. It is important that the check recognises soft 404s - that is, 200 responses whose content is effectively an error page - and resolves redirect chains to their final destination.
Crawled URLs: 2,418 | References checked: 9,732
GET / 200 ok
GET /products/current 200 ok
+ GET /products/legacy-item 404 dead page NEW
+ GET /blog/launch-2025 301 -> 404 NEW (redirect chain)
+ GET /category/accessories 200 soft 404 (empty) NEW
+ GET cdn.partner.io/widget.js timeout NEW (external link)
=> 3 new dead links + 1 dead external reference since deploy v54So that dynamically created links are not missed, the crawl should run in an environment that executes JavaScript - many modern sites build navigation and content only on the client side. Pure command-line fetches of the HTML source miss exactly these references. XICTRAQ website monitoring therefore checks pages in a real browser environment, follows internal and external links, and compares the result against the last known clean state.
What XICTRAQ Monitors for Links and 404
At XICTRAQ, link and 404 monitoring is not an isolated tool but part of a continuous target-versus-actual check of the entire website. The following building blocks interlock so that a new dead link does not surface only when ranking drops, but becomes a visible, alertable signal.
Regular crawl of all reachable pages, every internal reference is checked for its status code - new 404s stand out immediately.
Outgoing links to third parties are requested to capture timeouts, DNS errors and 4xx/5xx - even when nothing changed on your own page.
200 responses with effective error content are recognised instead of being treated as valid pages - the frequently overlooked case.
Redirects are resolved to the final target; chains that end in a 404 are reported as a dead link.
Missing images, scripts or stylesheets (404 on assets) are captured because they break the layout and make the page look broken.
Only genuine changes against the approved state raise an alert - via email, Slack, Discord or Telegram, rather than on every planned rebuild.
If the current state deviates from the approved one - a new dead link appears, a previously valid page tips into a soft 404, or a redirect chain breaks - it triggers an alert. That way you learn about a problem while it is still only a few hours old. The same principle of continuous monitoring applies to JavaScript error monitoring in the frontend, which surfaces client-side errors that a pure status-code check does not catch.
When you deliberately remove pages or change URLs and redirect them correctly, the link structure changes as planned. Re-approve the new target state consciously after each planned migration - then the monitoring only alerts on genuine, unintended dead links rather than on every planned change.
404 Monitoring as Part of Healthy Website Operations
Monitoring dead links is not an end in itself but a building block of reliable website operations. Reachability, correct status codes and intact references are part of basic technical hygiene - much like valid certificates, clean redirects and monitored API responses. Anyone who watches all these signals continuously detects problems as a change from the normal state rather than discovering them by chance.
The practical value lies in the time factor. An annual SEO audit records what the page looked like at one point in time - it says nothing about what happens at the next deploy. Continuous monitoring inverts this logic: it is not the one-off state that counts, but the change relative to the approved state. This makes monitoring compatible with modern development processes, in which the page is constantly changing anyway. Related target-versus-actual checks are covered in the articles on monitoring the CORS configuration and on monitoring API rate limits and 429 responses.
A deploy cannot be prevented - and it should not be. Fast releases are a competitive advantage. What can be prevented is a dead link created in the process silently costing ranking and conversions for weeks. That is exactly what automated link and 404 monitoring is for: it turns the invisible dead reference into a visible, alertable signal - before visitors and search engines stumble over it.
This article is based on data from: Pew Research Center (When Online Content Disappears, 2024 - 38% of webpages vanished over a decade, share of affected news and government pages), Ahrefs (link rot study - at least 66.5% dead links over roughly nine years), Semrush (on-site SEO study - 35% of sites with broken internal links, around 70% of them with a 4xx code), arXiv study 2024 (Mouchoux/Sehrawat, Quantification and Modeling of Broken Links Prevalence in Hyper Traffic Websites Homepages - 35.2% of the 88,000 most-visited homepages with at least one broken link), Google Search Central and John Mueller (documentation on 404, soft 404 and crawl budget; 404 is not a direct quality/ranking signal), Amazon Web Services / Forrester (88% no return after a poor experience), Baymard Institute (average cart abandonment rate ~70%), DORA State of DevOps 2024 (deploy frequency).