Monitoring Coverage Audit: The Pages You Forget

Green Means Checked, Not Healthy

A check target can be in one of three states, but a dashboard knows two colours. State one: checked and fine. State two: checked and broken. State three: not checked at all. The first two get a colour, the third gets no tile, and that is exactly why it looks like the first. An estate that consists of state three for ninety percent of its addresses shows an immaculate surface.

The status code itself says less than it is credited with. According to HTTP semantics, 200 indicates that the request has succeeded (RFC 9110) - not that the page delivered is the right one, is current, or contains what it is supposed to contain. And 404 indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists (RFC 9110); whether that condition is temporary or permanent is deliberately left open. Translating the number 200 into the words "all fine" already builds in an interpretation the protocol does not carry.

The HTTP Archive Web Almanac puts the same limitation very plainly, using robots.txt as its example: a 200 response only confirms that a file exists and says nothing about whether its directives are correct (Web Almanac 2025). That distinction carries over to every check target in a monitoring setup. Reachability is the weakest statement that can be made about an address, and in many installations it is the only one collected at all.

The Robots Exclusion Protocol states the problem most sharply. If the robots.txt file is unreachable because of server or network errors, it counts as undefined and the crawler must assume a complete disallow (RFC 9309). A failed measurement is therefore not a neutral state there but carries a defined, hard consequence. Monitoring setups usually lack that definition: when a run fails there is either no data point at all or one the interface treats like a passing check.

What a URL Inventory Is

A URL inventory is the complete, de-duplicated list of every address you consider yourself responsible for - each one annotated with where it came from, which status code it last returned, whether it is the canonical version and whether a check target points at it. That sounds like bookkeeping, and it is. The difference from the page list in your content management system: the content management system only knows what it produced itself.

What it does not know is plentiful: the redirect somebody entered into the web server by hand two years ago; the PDF price list in the upload directory; the booking flow of an external system that runs under your domain; the old campaign directory reachable only from an advertisement; the product feed a partner collects every hour. None of these addresses appear in a menu or a sitemap - but when one of them fails, somebody picks up the phone.

An inventory therefore has two properties a page list does not. It spans sources, and it remembers where each address came from. Provenance is not decoration: an address that appears only in the server log is a different case from one that sits in sitemap, crawl and search index at the same time. The first case calls for a decision, the second is uncontroversially business-critical and belongs in the check plan without further debate.

Four Sources, Four Blind Spots

In practice four sources are available. Each delivers a genuine section of the whole, and each has a limit that no amount of diligence removes, because it follows from how the source is built.

The sitemap is the only one of the four sources where you are the author. That is precisely why it is the most reliable in form and the weakest in substance: it contains whatever the generator writes into it, and the generator shares the limits of the content management system running it.

The server log is built the other way round. It claims nothing, it records. In exchange it only sees the past, and only the window your retention policy keeps open. A page that failed to attract a single request in four weeks does not exist in that source, even though it would be delivered if somebody asked for it. For maintenance, legal and archive pages that is the normal case.

A Sitemap Is a Declaration, Not a Finding

The sitemap protocol is clear about what it promises and clearer still about what it does not. A sitemap helps search engines discover addresses on your site but does not ensure that every item listed in it will be crawled and indexed (Google Search Central). It is a proposal, not a contract. For the inventory that means: what stands in the sitemap is a statement by the operator about their own estate, not a finding about what is actually delivered.

On top of that come hard formal limits that regularly break inventories apart without anyone noticing:

• Ceiling per file. Each sitemap file may contain at most 50,000 addresses and must not exceed 50 MB uncompressed (sitemaps.org). Larger estates need several files and a sitemap index; without that index the remainder stays invisible.
• Protocol and host binding. All addresses in a sitemap must use the same protocol and reside on the same host as the sitemap itself (sitemaps.org). Addresses on a subdomain do not belong there and get discarded - a frequent reason why a help or booking area is missing from the inventory.
• Scope of the storage location. A sitemap covers the path area it sits in. Move it into a subdirectory and its scope shrinks along with it, without anything changing inside the file.
• Ageing without feedback. A generator that leaves the last valid file in place after an error keeps returning status 200 (RFC 9110). The inventory ages quietly while the check on the sitemap request stays green.

That last line is the core of the topic: a sitemap check that only looks at the status code tests the file, not its contents. What helps is comparing the address count and the most recent modification date against the previous run. What that looks like as a continuous check is described in the article on sitemap monitoring.

Server Log and Crawl Contradict Each Other

Put the address list from the server log next to the result of a crawl and the two rarely agree - and the difference is the most interesting part of the inventory. Addresses that appear only in the crawl are delivered but requested by nobody. Addresses that appear only in the log are requested but not reachable through the website.

The second case is the more dangerous one because it runs out of sight. It covers addresses from old advertisements, from printed material, from a hard-wired partner interface and from the bookmarks of regular customers. They sit in no sitemap and no menu yet produce real traffic. When one of them drops to 404, monitoring reports nothing, because it does not know the address at all. The pattern resembles that of orphan pages, only inverted: there the incoming link is missing, here the entry in the check plan.

The crawl has two limits of its own. It follows only what stands in the delivered markup as a link; an address that comes into being after a click through a script stays hidden. And it obeys robots.txt, whose parsing limit must be at least 500 kibibytes (RFC 9309): whatever sits beyond that boundary in a very large file goes unread by conforming crawlers. In practice that rarely bites, since 97.5 percent of files on desktop and 97.6 percent on mobile stay below 100 bytes (Web Almanac 2025). More remarkable is the share of 1.8 percent completely empty files, because the standard defines no behaviour for the empty file (Web Almanac 2025).

Cached rules add to this. Crawlers should not use a cached robots.txt for more than 24 hours unless the file is unreachable (RFC 9309). A crawl applying yesterday's disallow rules produces yesterday's inventory. Change the file and expect a result the same day and you may still be measuring the old state. The most common mistakes of this kind are collected in the article on robots.txt errors.

Union, Normalisation, Canonicalisation

Before the four lists fit together they have to be brought into the same notation. Without that step half the difference between the sources consists of formalities: protocol, spelling of the host, prefix with or without www, trailing slash, order and presence of parameters, fragment identifiers, percent-encoded characters. Four lists of 1,200 addresses each quickly produce 3,000 entries under a naive union, 1,800 of which mean the same page.

The second step is canonicalisation. A canonical link names which version of a page is the authoritative one. It is present on 68 percent of desktop pages and 67 percent of mobile pages; 7 percent of desktop and 9 percent of mobile pages are canonicalised themselves, meaning they point at an address other than their own (Web Almanac 2025). For the inventory that means roughly two thirds of your addresses already carry the answer to the de-duplication question with them.

The remaining third and the contradictions stay manual work. Contradictions occur measurably: canonical mismatches are detected on 0.7 percent of pages (Web Almanac 2025). That sounds small but hits around seventy pages in an estate of ten thousand, and they are rarely the unimportant ones, because such conflicts arise where several systems deliver the same page. For the check plan the rule is: monitor the canonical version, give the variants a pointer to it and no entry of their own.

Equally important is how status codes are handled in the inventory. An address that redirects permanently belongs in the list as a redirect, not as a page; its check target is the redirect destination together with the redirect itself. The most unpleasant case is an address that serves an error message with status 200 - it counts as healthy in every statistic. How to spot that is covered in the article on soft 404 pages.

From the List to a Coverage Rate

An inventory without a metric stays a file. The metric that follows from it is the coverage rate: the share of addresses in the inventory for which an active check target exists and for which a result actually arrives within the expected interval. Both conditions count. A check target that has produced no result for eleven days covers nothing.

In practice three rates are carried side by side, because they expose different faults:

1. Raw coverage. Check targets divided by inventory size. Shows how large the unchecked remainder actually is.
2. Weighted coverage. The same calculation, weighted by requests or revenue contribution from the server log. Shows whether the checked addresses are the important ones - a raw rate of 20 percent can mean a weighted rate of 95 percent, and the other way round.
3. Effective coverage. Check targets with a fresh result divided by inventory size. Shows the failure of the measurement itself and is the only one of the three rates that drops when a check node stalls.

The third rate is the one missing from everyday practice. It answers not the question "Is the website fine?" but the question "Do I know whether the website is fine?". How closely that depends on the check interval is covered in the article on check intervals and detection time; which metrics belong on an overview is covered in the article on the monitoring dashboard.

Which Addresses Are Typically Missing

The gaps look remarkably similar across websites. Almost all of them are addresses nobody clicks while everything runs - and that is exactly why nobody thinks of them while setting monitoring up.

One pattern connects those six points: the failure goes unnoticed at first but has consequences - legal, contractual or financial. That is exactly what an inventory is for. For flows spanning several pages a status code is not enough anyway; those call for synthetic browser checks, and background work without an address of its own calls for heartbeats.

When a Failed Measurement Counts as a Result

So far this has been about addresses without any check target. The second and harder case concerns check targets that do exist but return no result - and whose silence gets read as calm.

The causes repeat themselves. The check node stalls, but its failure is not monitored. A timeout gets booked as "nothing found" instead of as a state of its own; in the Almanac it is a declared category and occurs on 1.0 percent of desktop and 1.1 percent of mobile robots.txt requests (Web Almanac 2025). A certificate error tears down the connection before a status code comes into being, and the result falls out of the evaluation. A disallow rule in robots.txt keeps your own crawler away from part of the estate, and that part shows up as "nothing found" instead of "not checked".

The Robots protocol shows what the clean separation looks like: it explicitly distinguishes between a file reported as unavailable through a status code and one that is unreachable. In the first case a crawler may access any resources, in the second it must assume a complete disallow (RFC 9309). That is a deliberate, written-out decision about how a failed measurement is to be treated. Monitoring needs the same decision, and it needs it visibly.

In practice that means three states in storage instead of two, a dedicated alert rule for absent results and a counter for the runs that actually took place. A website that has received no check for six days must look different from one that has stayed without findings for six days. The same distinction decides whether alerts deserve trust; how to set them so they do not go stale is covered in the article on alert configuration.

Keeping the Inventory Current

An inventory built once and then filed away is wrong again a quarter later. Maintenance consists of four recurring steps, all of which can be automated end to end.

The last point is the most effective. Comparing the inventory before and after a change catches precisely the class of fault rebuilds create: addresses that no longer exist afterwards without a redirect having been set up. The procedure for that is described in the article on migration monitoring.

While extending the inventory it helps to recall one basic fact: there is no central registry of all web pages, which is why search engines have to keep looking for new and updated pages (Google Search Central). What applies to search engines applies to you within your own estate just as much - the list does not appear on its own, it gets built and kept up.

The Next Step

Do not start with the tool, start with the count. Export the four lists once by hand, normalise them, form the union and count how many addresses it holds. Compare that number with the number of your active check targets. The ratio of the two is your raw coverage rate, and in most cases it turns out considerably lower than expected.

After that the order is simple: first take in the addresses carrying traffic, then the legally required ones, then the flows with several steps, and the peripheral areas last. Anyone already evaluating user paths and page transitions has half the data basis in the house. For the technical check targets that come out of this there is API monitoring and the status page with reporting; what each plan includes is shown in the pricing overview.