Soft 404s: When Error Pages Return HTTP Status 200

Why a wrong status code does damage longer than a short outage

An outage is unpleasant but honest. It gets measured, it raises an alert, it ends. A wrong status code is the opposite: it persists because hardly anyone notices it. The reason lies in how responsibilities are split on the web. People read the visible content of a page, machines read the first line of the response. As long as both say the same thing, the arrangement works. The moment the status line reports success while the content says „page not found“, the two worlds drift apart permanently - without producing a single warning signal.

The consequences are measurable. Search engines initially treat a 200 response as indexable content. Google describes exactly this case in its documentation: a page that answers with a success code but whose content is empty or error-like gets classified as a soft 404 and consumes crawl budget without return (Google Search Central). That costs twice over - requests flow into pages with no value, and genuine content gets fetched less often. On the French domain of the case study mentioned above, daily crawl requests fell from 60,000 to 70,000 down to 20,000 to 30,000 while soft 404 reports piled up (Search Engine Land).

On the visitor side the same defect plays out differently, but no less expensively. Anyone who clicks through from a search to a supposedly existing product page and finds a notice instead of an offer bounces. In retail the willingness to abandon is high anyway: the average shopping cart abandonment rate sits at around 70 percent (Baymard Institute), and every additional dead end makes it worse. That such dead ends accumulate over time is well documented - 38 percent of the web pages reachable in 2013 were no longer available ten years later (Pew Research Center). Websites lose pages; the only question is which code they admit it with.

What the standard requires and what servers actually send

HTTP semantics are unambiguous here. The 2xx status class means the request was received, understood and successfully processed (RFC 9110). 404 Not Found states that no representation was found for the requested address, without saying whether that condition is permanent. 410 Gone goes one step further: the resource is permanently gone from that address, and the server knows it (RFC 9110). Google treats both codes as a removal signal and explicitly recommends them for pages that no longer exist and for which no comparable replacement is available (Google Search Central).

In practice the behaviour often deviates - usually not out of negligence, but because a framework, a reverse proxy or a delivery network brings its own error handling. The following overview contrasts what each condition calls for with what systems tend to send instead.

The last row is deliberately different. A search results page without hits is not technically an error - the request was processed correctly, the result simply happens to be empty. It only becomes a problem when such pages are indexable and appear in large numbers, for instance because every filter combination generates its own address. Here the status code is not wrong, the indexability is. The remedy therefore differs: not a 404 response, but deliberate control through meta directives and canonical addresses - plus a check that makes empty result lists visible in the first place.

The four status code mistakes that occur most often in operations

Knowing the causes makes finding the affected addresses considerably faster. Four patterns typically account for the bulk of the findings.

Why the availability check does not see the defect

A classic availability check asks exactly one question: does a response arrive within the time limit, and is its status code below 400? A soft 404 satisfies both. The check reports green, the uptime figure stays spotless, and the customer report shows nothing unusual. That is precisely what makes this defect so durable: it survives every control that reads only the status line.

For applications that assemble page content in the browser, the situation gets worse. The server delivers a shell with 200, the script then decides whether a product exists and displays an error text if it does not. A plain fetch without execution sees none of that - neither the error text nor the empty result list. Only a check that actually renders the page brings the difference to light. That is what browser-based checks are for: they evaluate the visible state after rendering rather than just the raw response.

Detecting soft 404s through content signatures instead of the status code

When the status code lies, the check needs a second, independent source. That source is the content itself. A content signature is nothing more than an expectation about the response body: which strings must appear, which must not, and how large the response may be. Combined with the expected status code, this produces a statement that cannot be undermined by a wrongly set header line. Six signals have proven useful:

1. Expected status code per address: Store the specific expected code rather than „below 400“. An existing page expects 200, a deliberately removed address expects 404 or 410, a protected resource expects 401. Any deviation is a finding, including deviations upwards.
2. Forbidden text patterns at status 200: Phrases such as „not found“, „error“, „temporarily unavailable“ or „no results“ must not appear in a success response. A match is the most reliable indication of a soft 404 because it works independently of the response header.
3. Expected text patterns as a positive probe: Conversely, a product page should contain a price, an article number or an order button. If that marker is missing at status 200, the page is empty in content terms even though it responds technically.
4. A jump in response length: Error pages are typically shorter than genuine content. A sudden drop in response size across many addresses points to a centrally delivered substitute page - a pattern that a single spot check will rarely reveal.
5. A rendered check in the browser: For applications built on the client side, the page has to be executed before the signature applies. The browser check evaluates the state a visitor actually sees, including content loaded afterwards.
6. A counter-probe with an invented address: Regularly fetch a path that certainly does not exist, for example /this-path-does-not-exist. If it answers with 200, the error handling of the entire website is misconfigured - regardless of how many real addresses happen to be clean at that moment.

The counter-probe from the last point is the fastest single test there is, and it is worth running before any larger change. It also exposes an adjacent case that unfolds just as quietly: pages that respond correctly but serve outdated content. Why this happens particularly often after a release is described in the article on stale caches after a deploy and outdated content.

Labelling maintenance, throttling and legal blocks correctly

Not every special condition is a fault, but every special condition has a matching code. For planned work that code is 503 Service Unavailable: the service exists but is temporarily unreachable. What matters is the accompanying Retry-After header, which names either a number of seconds or a specific point in time (RFC 9110). Google names a clear upper limit: as short-term throttling, 503 or 429 are meant for one to two days; if the crawler sees those codes on the same URL for multiple days, it may be dropped from the index (Google Search Central). A maintenance page with a success code, by contrast, gets handled as regular content - in the worst case the sentence „we will be back shortly“ ends up as the description in a search result. How to separate planned work cleanly from a real incident is covered in the article on maintenance windows and false alerts in monitoring.

The same logic applies to throttling. 429 Too Many Requests states that the client has exhausted its quota and should try again later (RFC 6585); here too a Retry-After belongs in the response. Answering with 403 instead claims a missing permission, and answering with 200 claims success - both mislead the client. A special case is 451 Unavailable For Legal Reasons: the code marks content that may not be delivered for legal reasons and was standardised specifically for that purpose (RFC 7725). It replaces no legal assessment, but it makes the block traceable instead of hiding it behind a 404. If 5xx or 429 responses persist over a longer period, affected addresses can drop out of the index (Google Search Central) - so a temporary code must not become a permanent state.

Spotting status code drift after a deploy early

Wrong status codes rarely appear on day one. They appear with a release that changes a routing rule, with a new rewrite rule in the web server, with a rule in the delivery network, or when the framework is swapped. Yesterday /products/item-4711 still answered with 404, today it answers with 200 and a generic page. That transition is exactly what status code drift means, and it is easy to measure - provided someone compares the response against the previous state. The graphic above shows precisely that sequence: four checks with a correct 404, then a release, then 200.

The table makes clear that no single type of check suffices. Only the combination of expected code, content signature and rendered state covers the field. A look at the presentation layer helps as well: a visual comparison of page states shows when a designed error page suddenly turns into an empty surface. And because not every disruption becomes visible over HTTP, the reachability of the underlying services belongs in the picture too - the article on port monitoring for services beyond the web page describes how to secure that layer.

A check plan for correct status codes

The preceding points turn into a routine that can be set up in a single morning and then runs without further effort.

• Take stock: Fetch an invented address, a deliberately removed address and a genuine existing page, and note the actual status codes. Three requests already show whether the error handling is fundamentally sound.
• Write down the expectations: Define the expected code for each monitored address instead of accepting everything below 400 across the board.
• Store forbidden patterns: Define a short list of error phrases that must not appear in a 200 response - matched to your own interface and its wording.
• Add a positive probe: For the most important templates, store a marker that has to be present, such as the article number on product pages or the submit field in a form.
• Verify the maintenance response: During the next planned window, test whether 503 with Retry-After is really delivered - and not 200 with a notice.
• Rendered checks for dynamic pages: For areas built on the client side, set up a browser check that evaluates the visible end state.
• Log the changes: Use screenshots and an automatic change log to record how error and maintenance pages looked at any given point - that shortens root cause analysis after a release considerably.
• Alert on deviation, not on outage: Configure notifications so that a status code deviation already reports - not only the complete outage.

A note on the reports inside search engine tooling: the soft 404 reports there are a useful supplement, but no substitute for your own checks. Their reporting accuracy was revised at one point, which is why older figures are not directly comparable with newer ones (Google Search Central); the classification is also made on the mobile version of a page (Google / John Mueller). Above all, though, those reports only arrive after the crawl - your own check reports within minutes.

Checking status codes reliably with XICTRAQ

A soft 404 is not an exotic edge case but the predictable result of a check that reads only one line of the response. As soon as the expectation about the status code is spelled out and complemented by a statement about the content, the blind spot disappears - and an invisible permanent condition becomes a finding with a timestamp.

That is exactly what availability monitoring from XICTRAQ is built for: every check can be pinned to an expected status code, complemented by required and forbidden text patterns in the response body. For applications that only finish in the browser, rendered checks evaluate the visible end state. For interfaces, API monitoring applies the same logic at field level, so a response with status 200 and an empty result object stands out too. Visual comparison and the change log record when the behaviour changed, and alerting fires on the deviation itself.

Use the free trial to check which status code your error page really sends today - fetching an invented address takes a few seconds. If you would rather walk through the setup of content assertions and rendered checks together, book a personal demo.