Why a partial outage stays hidden for so long
Ordinary monitoring answers a single question: does the address respond? Ask that question from one point and you get an answer along exactly one path through the network. If that path is fine, the check reports green while another part of the internet cannot reach the same address at all. The fault is therefore not invisible, it is merely measured in the wrong place. Why a single vantage point is too little for this class of disruption is set out in Multi-location monitoring: why one location is not enough; for routing faults it applies in sharper form, because such a disruption is location-dependent by definition.
The second reason for the late discovery lies in how complaints behave. Affected visitors sit behind a particular access network, often in a particular region. They rarely get in touch, and when they do, their description sounds like a problem at their end. Measure from several networks and you see a pattern instead: the same address, the same second, different results per vantage point. Separating that pattern from genuine noise is a task of its own; how to tame recurring up-and-down notifications without losing the real ones is covered in Taming flapping alerts.
- By prefix count, 55.8 per cent of routed IPv4 prefixes and 60.4 per cent of routed IPv6 prefixes were covered by a signed statement of origin (ACM IMC 2025).
- By address space, the figures were 51.5 per cent for IPv4 and 61.7 per cent for IPv6 as of April 2025 (ACM IMC 2025).
- 49.3 per cent of organisations holding their own address allocation had issued at least one ROA in early 2025, and 44.9 per cent had done so for all of their address space (ACM IMC 2025).
- In the RIPE region, almost 80 per cent of routed IPv4 address space was covered by April 2025 (ACM IMC 2025).
- Less than 5 per cent of prefixes classified as RPKI-Invalid reached a visibility of more than 40 per cent of the route collectors, while more than 90 per cent of valid and not-found prefixes were seen by more than 80 per cent of the collectors (ACM IMC 2025).
These figures describe the state of global routing, not the state of a single website. They are still the basis for a practical assessment: 40 to 50 per cent of the routed address space remains uncovered by ROAs (ACM IMC 2025). For a prefix without such a statement, a wrong announcement is not flagged automatically anywhere; for a prefix with one, many large networks discard a conflicting announcement while others accept it. That is precisely what produces the picture of a service that is only partly reachable.
Prefix, autonomous system, announcement
The internet is not a single network but an association of independent ones. Each of those networks carries a number, the autonomous system number, and is responsible for the reachability of one or more address ranges. Such an address range is called a prefix and is written as an address with a length, for example 203.0.113.0/24. Through the Border Gateway Protocol a network tells its neighbours which prefixes it makes reachable and along which chain of networks the path leads there. That message is the announcement, and it underpins every connection that crosses more than one network. RFC 7454 describes the protocol as the practically exclusive means by which networks on the internet exchange routing information.
The point that matters operationally: there is no central list against which an announcement would be checked. Every network decides for itself which announcement it accepts and which it passes on. A network can therefore know a different path from its neighbour, and both can be right. The same property makes the internet robust and troubleshooting laborious. It also explains why a service can be reachable over IPv4 and not over IPv6, or the other way round: both address families are announced separately and assessed separately. Monitoring both paths finds these cases; what that looks like is described in Monitoring IPv6 reachability in a dual stack.
Prefix hijacking is the unauthorised announcement of an IP prefix that is assigned to a different autonomous system.
— BSI, Cyber Security Publications, How-To: RPKI (translated from the German original)
How a wrong announcement comes about
The most common cause is not an attack but a typo or a forgotten line in a neighbouring network's configuration. One digit too many turns into a prefix allocated to somebody else; a missing filter rule turns into an announcement that was only ever meant to apply internally. Such mistakes travel through large parts of the internet within minutes and are, for the party affected, initially indistinguishable from a deliberate takeover. That attacks on name resolution produce a similar picture does not make classification easier; the differences are described in Detecting and preventing DNS hijacking.
- A more specific announcement: another network announces a smaller slice of the prefix. Because the more specific entry usually wins, it pulls the traffic towards itself without the original announcement disappearing.
- A wrong origin AS: the prefix is announced correctly but under a number that is not authorised for it. Without a signed statement of origin, nothing flags this automatically.
- Propagation beyond the boundary: a network passes on announcements it received for its own use only. Traffic then takes a detour across a link that is not dimensioned for that volume.
- A withdrawn announcement: after a migration the announcement for part of the address range is missing. The rest stays reachable, the missing part disappears silently.
- A filter at the neighbour: a correct announcement is discarded by a single network, for instance because an outdated filter list sits there. Exactly one access network is then affected.
For the effect, the cause is secondary at first. Typo, stale filter or intent: the path to part of your audience is missing while your own infrastructure keeps running without a sign. What this shares with other silently failing layers is the absence of feedback, because anything that does not disturb ordinary requests goes unnoticed in operations for a long time. An open zone transfer in DNS belongs to the same family; how to find one is covered in Open zone transfer via AXFR.
When two of eight vantage points cannot reach an address and the other six answer in milliseconds, that is not a measurement error to be smoothed away. A consensus rule that only reports once a majority of points fails suppresses exactly this case. It is more useful to add a second rule next to the consensus: alert as well when one vantage point deviates across several rounds while the others answer steadily.
RPKI: the signed statement of origin
The Resource Public Key Infrastructure closes exactly one gap: it records, cryptographically signed, which autonomous system may announce a given prefix and up to which length. That object is called a Route Origin Authorisation, or ROA. It contains the authorised autonomous system, the prefix and the maximum permitted prefix length. A network that validates origin compares every incoming announcement against the recorded statements and arrives at one of three results. The procedure is set out in RFC 6811, the operational recommendations in RFC 7454.
| State of the announcement | What it means | What networks do with it |
|---|---|---|
| NotFound | No recorded statement of origin covers this prefix | The announcement is usually accepted as before |
| Valid | A statement of origin matches prefix, length and origin AS | The announcement counts as confirmed |
| Invalid | A statement covers the prefix but does not match the AS or the length | Networks that validate origin typically discard the announcement |
What matters is what follows from the result and what does not. RFC 6811 defines the three states but leaves the decision to the individual network: an announcement is not dropped from processing automatically because of its state unless that has been configured explicitly. In practice many large transit networks now discard invalid announcements while smaller networks often do not. That is exactly what creates the divided view: less than 5 per cent of invalid prefixes reached a visibility of more than 40 per cent of the route collectors in the measurement, while more than 90 per cent of valid and not-found prefixes were seen by over 80 per cent of the collectors (ACM IMC 2025). A wrong announcement is therefore not believed everywhere, but it is not discarded everywhere either.
How far coverage reaches today
As of April 2025, 51.5 per cent of routed IPv4 address space and 61.7 per cent of routed IPv6 address space were covered by ROA certificates; by prefix count the figures were 55.8 per cent for IPv4 and 60.4 per cent for IPv6 (ACM IMC 2025). At the level of organisations the picture looks similar: in early 2025, 49.3 per cent of organisations holding their own address allocation had issued at least one ROA, and 44.9 per cent had done so for all of their address space (ACM IMC 2025). The distribution across regions is uneven. In the RIPE region, which includes Germany, almost 80 per cent of routed IPv4 address space was covered by April 2025 (ACM IMC 2025).
The remainder is an organisational task more than a technical one. The same study classifies 47 per cent of IPv4 prefixes and 71 per cent of IPv6 prefixes without an RPKI record as coverable with minimal technical effort (ACM IMC 2025). It also calculates that global coverage could rise by 7 per cent for IPv4 and 19 per cent for IPv6 if as few as ten organisations took the necessary steps (ACM IMC 2025). For a website operator this means one thing above all: the question to the hosting provider is legitimate and can usually be answered without much effort.
The shares by address space and prefix count come from a measurement of routed prefixes that evaluates the public route collectors of RouteViews and RIPE RIS and excludes prefixes more specific than /24 for IPv4 and /48 for IPv6 as well as prefixes seen by fewer than 1 per cent of the collectors; the reference date is April 2025. The organisational shares, by contrast, rest on the bulk WHOIS holdings of five RIRs and three NIRs, from which the study determines direct allocations via the prefix hierarchy; there the reference date is early 2025 (ACM IMC 2025). They therefore describe the global picture, not the situation of a single network. For your own address, all that counts is whether a matching statement of origin exists for the prefix under which the website is reachable.
maxLength: the switch that permits too much
Alongside the prefix, a ROA can carry a maximum length. This allows the authorised autonomous system to announce smaller slices of the range as well, without needing a separate object for each slice. That is convenient, and it has a side effect: every slice that is permitted but not announced sits there as an admissible entry with no real announcement occupying it. Anyone who announces it under the authorised number produces an announcement that passes origin validation and pulls the traffic towards itself.
RFC 9319 examined how widespread this is. A measurement from June 2017 found that 12 per cent of the prefixes authorised in ROAs carry a maximum length longer than their own prefix length (RFC 9319). Of those, 84 per cent were non-minimal because they included slices that the legitimate autonomous system does not announce at all (RFC 9319). The document's recommendation is correspondingly brief: use minimal ROAs wherever possible, and set the maximum length only where the smaller slices are actually announced. For a conversation with a hosting provider, that is a concrete and verifiable question.
Detecting and classifying a partial outage
The first step is unspectacular: query the same address at the same time from several points and lay the results side by side. Only the comparison turns four individual measurements into a statement. How closely the rounds need to sit depends on the detection time you want; the arithmetic for that is in Check intervals and detection time. For routing disruptions a second requirement is added: the vantage points should sit in different networks, not merely in different places. Two data centres on the same upstream see the same announcement and therefore return the same answer.
The four calls separate four layers. Name resolution returns an address, the connection does not come up, the path breaks off after the ninth hop, and the far end does not answer a simple echo either. The return value 000 on the second line matters here: that is not a status code but the statement that no answer arrived. A value produced by a failed request does not belong in the same field as a genuine measurement, otherwise a failure of the measurement turns into a measurement result. The same sequence from a point in a different network returns a complete picture with status code 200 during a partial outage.
Uptime monitoring queries the same address from several EU locations and forms a consensus from the individual results. If one location deviates persistently, that is a finding of its own rather than noise.
DNS and domain monitoring compares the answers of several resolvers. That makes it possible to tell early whether the address is wrong or the path to the correct address is missing.
API monitoring checks connected services separately from the website. If an interface fails from one direction only, the direction is in the log.
Alerting states in the body of the notification which vantage points are affected. A notification without a location costs more time in triage than the check saved.
What is possible without your own network
Anyone running a website usually holds neither an autonomous system nor address ranges of their own. The addresses belong to the hosting provider, and so the statements of origin are that provider's responsibility too. You can still act, in three places: at selection, at measurement and at reporting. The questions below can be asked before signing a contract and repeated later. They are understandable without networking knowledge and have a short, verifiable answer.
- Are statements of origin on file for the prefixes our addresses come from? The answer is yes or no and can be checked independently.
- Is the maximum length used sparingly? The question is whether the objects permit only those slices that are actually announced.
- Does the provider validate the origin of incoming announcements? This concerns the path in the other direction and affects whether your own visitors can be steered onto a wrong path.
- Is there a named route for routing reports? An address that a suspicion with a measurement log can go to is worth more in an emergency than a general fault hotline.
- How is a regional disruption communicated? A status message that names the affected area is more useful than reporting a disruption either in full or not at all.
- Are IPv4 and IPv6 covered equally? Both address families are announced separately; a statement for IPv4 only leaves the second half open.
The time window with its time zone, the affected vantage points with their network numbers, and the unaltered output of the measurement. If one of them is missing, the recipient starts with a follow-up question and the suspicion ages. Being able to pull these three details out of the running log typically means reporting in minutes rather than hours.
Documenting a suspicion properly
A report that consists of a hunch produces a follow-up question at the provider and little else. It becomes solid through a log with three properties: it names the vantage points individually, it distinguishes between a measured value and a failed measurement, and it carries timestamps in a zone the recipient understands too. The last point sounds incidental and is one of the most common sources of misunderstanding in fault reports.
{
"address": "https://www.example.invalid/",
"prefix": "203.0.113.0/24",
"observed_from": "2026-09-24T08:12:00Z",
"observed_until": "2026-09-24T09:41:00Z",
"vantage_points": [
{ "site": "Frankfurt", "network": "AS64496", "state": "reachable", "http": 200, "ms": 41 },
{ "site": "Amsterdam", "network": "AS64497", "state": "reachable", "http": 200, "ms": 53 },
{ "site": "Madrid", "network": "AS64498", "state": "no route", "http": null, "ms": null },
{ "site": "Milan", "network": "AS64499", "state": "no route", "http": null, "ms": null }
],
"consensus": "partially reachable",
"dns_answer_identical": true,
"failed_measurement_as_zero": false
}Two details in this log carry the burden of proof. First, the unreachable locations do not hold zero values but empty fields with a note that no route existed; a 0 in the response-time field would be an assertion about something that was not measured at all. Second, the timestamps are recorded in UTC. Work with local times in autumn and you get an hour that occurs twice, and with it an order of events that can no longer be reconstructed; what follows from that for monitoring is set out in Daylight saving time in monitoring.
A log of vantage points shows that an address was unreachable from certain networks. It does not show why. Attributing it to a particular announcement, a particular network or even an intention is a statement about third parties and does not belong in a report without evidence. It is better phrased as an observation: time window, vantage points, result, request for clarification. The root-cause analysis sits with whoever can see the announcements.
Alerts and status page for a regional disruption
An alert for a partial outage needs a different threshold from one for a complete outage. A rule that knows two cases is useful: if the majority of vantage points fail, that is an outage; if one vantage point deviates across several rounds while the others answer steadily, that is a regional disruption with a notification of its own. Both notifications need the location in the text, otherwise triage at the recipient's end starts with a search. Externally, the same distinction belongs on the status page: a public status report that names a regionally limited incident as such saves a share of the enquiries and is more honest than a green light.
For the history, recording per location pays off. Only that answers the question that regularly follows an incident: has this happened before, and did it hit the same networks? A history kept per vantage point shows repetitions that vanish inside an aggregated availability figure. The same figure can mean that every visitor was affected briefly, or that two of eight vantage points did not get through for hours. For the assessment that is a difference; for the metric it is not.
Bringing routing reachability into everyday operations
The routing layer cannot be steered from outside, but it can be observed, and being observable is half the way there. Measure from several networks, keep the results per location and record the difference between a measured value and a failed measurement, and you will typically spot a partial outage in the same hour rather than the week after next. The rest is communication: a report with three details, a named route to the provider and a status page that names regional disruptions. That a provider makes announced changes visible in good time belongs to the same expectation; how to measure that for interfaces is shown in Monitoring Sunset and Deprecation headers. If you would like to see the setup first without configuring anything, the demo view shows the picture across several locations.
- Vantage points set up in at least three different networks, not merely in three places
- IPv4 and IPv6 checked separately and evaluated separately
- Consensus rule extended by a second rule for individual points that deviate persistently
- Failed measurement and measured value distinguishable in the log, no zero values for anything unmeasured
- Timestamps kept in UTC, local time only in the display
- A named contact route at the hosting provider for routing reports, tried out once
- Status page knows the state partially reachable
- Questions about statements of origin and maximum length asked in the last provider review
The share figures come from the study ru-RPKI-ready: the Road Left to Full ROA Adoption, published at the ACM Internet Measurement Conference 2025. The figures on maximum prefix length are in RFC 9319 (BCP 185). The three states of origin validation are defined in RFC 6811, the operational recommendations in RFC 7454 (BCP 194). The definition quoted is taken from the How-To RPKI in the BSI's Cyber Security Publications. All shares refer to the reference date stated with them and change continuously.