Origin Behind the CDN: When Cache Hides an Outage

What the green check mark actually proves

An HTTP check proves exactly one thing: someone answered at the address under test with a status code. Who sent that response is not part of the status code. If a CDN sits in front, the check typically terminates at the nearest edge, and the edge has two options - ask the origin or answer from its own store. The second path is faster and therefore the usual one. A 200 consequently says nothing about whether the application behind it is still running, whether the database is reachable, or whether the checkout completes.

This is not a malfunction of the cache, it is its job. RFC 9111 describes a stale response as one that carries expiration information or may be calculated heuristically, but is no longer fresh after the freshness calculation. A cache may generate it anyway, but only under one of two conditions: it is disconnected from the origin, or the client or the origin server explicitly permitted it, for instance through the extensions in RFC 5861. Where a directive such as no-cache, must-revalidate, s-maxage or proxy-revalidate prohibits the stale response, it stays forbidden in those situations too (RFC 9111). So the edge does what it was told to do - it keeps the visitor away from an error page. But in doing so it also keeps the monitoring away from the situation.

• What the check proves: the address is reachable, TLS is up, a status code came back, the response time was within range.
• What the check leaves open: whether the origin server was involved in that response and how old the delivered content is.
• What follows from that: an origin outage stays invisible for as long as the edge has supplies - with an unfavourable setting, for hours.
• What that means for detection time: the chosen check interval shortens nothing here, because every single check receives the same cached response.

Age: the header that reveals the origin

There is one signal available without any extra effort that answers half the question: the Age header. RFC 9111 states its evidential value in two sentences that have to be read separately. The first: the presence of an Age header field implies that the response was not generated or validated by the origin server for this request (RFC 9111). The second limits the inverse: a lack of an Age header field does not imply the origin was contacted (RFC 9111). An Age value is therefore solid proof of a cache response, but its absence is no proof of the opposite.

In practice this means: an Age value that grows steadily across several checks while nothing else changes is the signature of a frozen copy. As long as the origin responds, the value drops back to a small number after each refresh. If instead it keeps climbing past the freshness lifetime, the edge is serving from its store - and that is the point at which monitoring that only reads the status line stops knowing anything. The value alone is not yet an alert, but it is the only figure in the standard set of response headers that points in this direction at all.

How long the blind window is

The duration of the blindness can be calculated as soon as two values are known: the freshness lifetime and the grace period for error cases. RFC 5861 introduces stale-if-error for this and defines the notion of an error narrowly - it means any situation that would result in a 500, 502, 503, or 504 status code being returned (RFC 5861, Informational). The example in the document is clear enough to be adopted directly: a response with max-age=600 and stale-if-error=1200 is fresh for 600 seconds and may then be used for another 1200 seconds in the error case. Once the age exceeds 1800 seconds, the cache must write the error message through (RFC 5861, Informational).

Two rows of this table deserve particular attention. First the heuristic: if a response carries neither an expiry date nor Cache-Control, a cache may estimate the freshness lifetime, and RFC 9111 names ten percent of the time since the last modification as a typical setting (RFC 9111). For a file last modified a year ago that is a good five weeks. Second the defaults: in mod_cache, serving stale content in place of 5xx responses is on by default (Apache project), whereas in nginx proxy_cache_use_stale ships as off (nginx). The same application therefore behaves fundamentally differently behind two intermediary layers - and both behaviours are documented, neither of them is a bug.

Cache-Status: the answer to the actual question

Since RFC 9211 there has been a standardised header that reports exactly what monitoring wants to know. The Cache-Status field carries a list in which each member represents a cache that has handled the request; the first member represents the cache closest to the origin server, and the last member represents the cache closest to the user (RFC 9211). This makes it possible to read how many layers were involved and what each of them did. For the case at hand, one sentence from the document is decisive: a stale response that is used without going forward - for example because the origin server is not available - can be considered a hit (RFC 9211).

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=600, stale-if-error=1200
Age: 1490
Cache-Status: upstream-cache; fwd=stale; fwd-status=503, edge; hit; ttl=-890

Three pieces of information in this response contradict the green check mark. Age stands at 1490 seconds and is therefore far beyond the 600 seconds of freshness lifetime. The ttl parameter states the remaining freshness and may be negative to indicate staleness (RFC 9211) - here 890 seconds are missing. And fwd=stale states that the cache was able to select a response for the request, but that it was stale (RFC 9211), while fwd-status carries the status code the next-hop server returned (RFC 9211). So the origin answered 503 and the visitor received 200. Both are in the same response, and a check that only reads the status line sees none of it.

How widespread the situation is

The mechanism is not a special case of individual setups. In the Web Almanac 2025, 35 percent of mobile HTML requests are answered via a CDN, subdomain resources reach 52 percent and third-party content 71 percent (Web Almanac 2025). For HTML the share rose from 33 to 35 percent between 2024 and 2025 (Web Almanac 2025) - so the proportion of sites with a layer between check and application is tending to grow. Anyone planning to use a CDN is planning that layer into their own monitoring as well.

The distribution explains why the problem so often goes unnoticed. Explicit instructions for the error case are the exception, the defaults of the intermediary layers are the rule - and those already differ, as the table shows, between two widely used servers. For monitoring, an uncomfortable consequence follows: the blind span of a website cannot be derived from the application, it can only be measured on the delivered response.

Two check paths instead of one

The resolution is unspectacular and costs little: check both ends of the chain separately and compare the results. The public path measures what the visitor receives. The second path goes past the cache and measures whether the application responds. Together the two results carry a statement that neither of them can carry alone.

• Public path: the regular address via the edge, with assertions on status code, content and response time.
• Origin path: the same application addressed directly, either through a dedicated hostname or through a request with a Host header set against the origin address.
• Health endpoint: a lean address that touches the database and the most important dependencies and is explicitly delivered as non-storable.
• Comparison: if the origin path deviates from the public path, that is the finding - and it arrives before the edge has used up its supply.

The first call answers the question of provenance without anything having to be installed. The second bypasses the edge through a fixed name resolution and hits the origin directly - here the 503 appears that the visitor does not get to see. The third checks whether the health endpoint is itself storable; if it is, that path too will eventually measure nothing but a copy of itself. Anyone checking from several locations additionally recognises whether only individual edges are affected - which is the subject of the article on monitoring from multiple locations.

Assertions that measure the difference

Two paths only turn into monitoring once rules are attached to them. What helps are assertions that do not repeat the status code but evaluate the provenance and the age of the response. Three rules cover the majority of cases: an upper bound for Age on the public path, an exclusion of fwd=stale across several consecutive checks, and a hard status comparison between the public path and the origin path.

from dataclasses import dataclass

ERROR_STATUS = {500, 502, 503, 504}   # notion of an error from RFC 5861

@dataclass
class Response:
    status: int
    age: int | None             # None means: the header was not set
    max_age: int | None
    stale_if_error: int | None
    fwd: str | None             # value from Cache-Status, if present

def blind_window(r: Response):
    """Seconds during which a dead origin can stay hidden."""
    if r.max_age is None:
        return None                     # unmeasured, not 0
    return r.max_age + (r.stale_if_error or 0)

def state(r: Response) -> str:
    if r.status in ERROR_STATUS:
        return "error written through"
    if r.fwd == "stale":
        return "origin not responding"
    if r.age is None:
        return "undetermined"           # no evidence in either direction
    if r.max_age is not None and r.age > r.max_age:
        return "stale copy"
    return "fresh"

def remaining(r: Response):
    window = blind_window(r)
    if window is None or r.age is None:
        return None
    return max(window - r.age, 0)

The functions deliberately return None when the Age header is missing or max-age is not set. A missing header is not a zero value, it is a measurement that did not happen - and monitoring that treats the two alike reports calm where it measured nothing. The same separation pays off for response times, where a single outlier disappears in the average and only becomes visible in the p95 and p99 percentiles.

Alerting before the supply runs out

The timing of the alert decides whether the monitoring is still useful. If the alert only fires once the edge has no copy left, the outage is already visible to everyone - at that point the monitoring reports the same thing as the telephone. The gain lies in the span before that: it begins the moment the origin path turns, and it ends as soon as the age of the delivered response exceeds the sum of freshness lifetime and grace period (RFC 5861, Informational).

• Stage 1, immediately: the origin path answers with 500, 502, 503 or 504 (RFC 5861, Informational) while the public path still answers 200. This is the silent phase and the earliest sensible moment for a notification.
• Stage 2, after confirmation: fwd=stale appears on the public path or ttl turns negative (RFC 9211). The edge is now demonstrably serving from its store.
• Stage 3, before the end: Age approaches the sum of max-age and stale-if-error. From here the outage is foreseeable for visitors and no longer only for the monitoring.
• Standing rule: the health endpoint stays uncached, otherwise stage 1 loses its basis. How to separate error responses cleanly from outliers is covered in the article on the 5xx server error rate.

A green check mark without a statement about the provenance of the response is a statement about the cache, not about the website. The difference costs almost nothing to set up and decides whether a notification arrives in time.

— XICTRAQ

What can be set up this week

The effort stays manageable because all the required information is already in the response headers. One pass: retrieve the Age, Cache-Control and Cache-Status headers on a sample page and note the blind span. Then set up the origin as a separate check path, either through an internal hostname or through a fixed name resolution. Finally, add an assertion on the Age value and set the health endpoint to no-store. A complete inventory of the checked addresses shows which entry points have no check path at all so far.

Ahead of traffic peaks the pass is worth twice as much, because the hit rate at the edge and the load at the origin rise at the same time - what to watch out for is covered in preparing for peak season. In XICTRAQ both paths can be run as separate checks, header assertions belong to API monitoring, and beyond the configured alert channels the notification also reaches the status page so that customers see the same situation as the team. Anyone who wants to look at the interplay first will find it in the demo view; the basics of availability checking are described under uptime monitoring.