What flapping alerts are — and why they destroy trust
A monitor works within a simple state model: each check returns a result that the detector translates into a state — typically up or down. When that state changes, the system fires a notification. With flapping, the state does not change once or twice but in rapid succession. Each change can trigger two messages: a problem message on the jump to down and a recovery message on the jump back to up. Ten changes in twenty minutes therefore produce up to twenty messages for a single, not-yet-understood incident.
It is exactly this combinatorics that makes flapping so dangerous to a team's attention. Nagios deliberately limits its flap-detection window to the last 21 check results per host or service — with 21 stored results there can be at most 20 state changes (Nagios Core Documentation). That ceiling is no accident but a formal acknowledgement of the problem: a single service can produce a surprising number of transitions in a short time, and each one competes for a human's response. The monitored service itself may not have truly failed even once during that window.
The Google SRE Book warns explicitly: "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page" (Google SRE Book). An inbox full of flapping messages is therefore not just irritating but a safety risk: the one alert that matters disappears among nineteen that did not.
The two causes: a threshold at the edge and an over-sensitive detector
Flapping almost always has the same twofold root. The first cause is a threshold at the edge: a measured value hovers right around the boundary where the detector flips. Response time sits just at the defined timeout, CPU load scratches the alarm line, a single server in the load-balancer pool sporadically drops out of rotation. The natural spread of such values is then enough to push the state above and below the boundary in turn — without anything fundamental changing about the actual service.
The monitored metric sits permanently close to the threshold. Measurement noise, jitter or normal load swings are enough to flip the state back and forth — a systematic problem of threshold choice, not of the service.
The detector decides on the basis of a single sample. One dropped packet, one slow response or a transient TCP reset is enough for an immediate down — no smoothing, no confirmation, no second opinion.
The second cause is an over-sensitive detector: it validates a state from a single measurement. The internet, however, is a statistical medium. Packet loss, brief latency spikes, a renegotiating TLS handshake or awkward routing between two networks are the normal condition, not an outage. Turning every single blip into an immediate down mistakes a snapshot for a diagnosis. Only the interplay of both causes — the metric at the edge plus the single-sample logic — produces the characteristic storm. Both can be addressed deliberately, and that is exactly what the following three remedies do.
What flapping costs: alert fatigue and the missed outage
The obvious damage is alert fatigue. The SRE Book describes the burden soberly: "If the employee is at home, a page interrupts their personal time, and perhaps even their sleep" (Google SRE Book). Every unnecessary interruption costs concentration, rest and, in the long run, the willingness to take alarms seriously at all. Google derives a clear standard from this: "Every page should be actionable" (Google SRE Book). A flapping message is the opposite of that — it demands a response without enabling a meaningful action, because by the time someone looks, the state has often already jumped on three more times.
The subtler, costlier damage is the missed real outage. Good monitoring, per the SRE Book, answers two questions: "what's broken, and why?" (Google SRE Book). To achieve that, Google recommends focusing on the four golden signals — latency, traffic, errors and saturation (Google SRE Book) — rather than on every small twitch of state. Where a team is used to flapping, the reflex is to click alerts away. And that is exactly when the next message, which is not a twitch but the beginning of a real outage, meets a desensitised eye. The SRE Book states the consequence bluntly: "If a page merely merits a robotic response, it shouldn't be a page" (Google SRE Book).
Flapping does not primarily cost compute or storage but attention — the scarcest resource in operations. Every message that fires without being actionable lowers the value of all future messages. The goal is therefore not more alerts but fewer, reliable ones. That is precisely what the three remedies aim at.
Remedy 1: require consecutive failures
The first and most effective remedy against the nervous detector is a simple rule: an outage counts as confirmed only when several consecutive checks fail. Instead of alerting on the first blip, the monitor requires, say, three failed checks in a row before it sets the state to down. A single lost request no longer triggers a message — precisely the transient blips that produce most of the noise are filtered out before they become an alert.
Nagios casts this idea into a mature formula. From the last 21 results, flap detection computes a percent state-change value and compares it against thresholds (Nagios Core Documentation). Newer changes weigh more heavily: "The flap detection routines are currently designed to make the newest possible state change carry 50% more weight than the oldest possible state change" (Nagios Core Documentation). An outage that has only just begun is therefore weighted higher than one that lay twenty checks ago — detection reacts to current instability, not to long-past events.
- Choose the count deliberately: Two to three consecutive failures filter out most single blips without noticeably delaying detection.
- Budget for detection latency: Each additional required failure pushes the alert back by one check interval. At a 60-second interval, three attempts mean up to three minutes of delay — a conscious trade between calm and reaction speed.
- Tier by criticality: A payment endpoint may require fewer confirmations than a marketing page. The count is a dial, not a dogma.
- Align with the check interval: Tight intervals allow more confirmations at the same latency; wide intervals call for restraint on the count.
This retry-before-alert logic is the foundation of any calm uptime monitoring. It costs a few seconds of detection latency and repays them as drastically fewer false alarms. How to configure whole alert chains cleanly beyond this is explored in the article on monitoring alerts and alarm fatigue.
Remedy 2: confirm from multiple locations
Consecutive failures filter temporal noise. But a second, often underestimated cause of false down is spatial: the monitor sees the world through a single network path. A peering problem between two providers, a congested transit link, a regional DNS hiccup or a brief routing anomaly can make the target site appear unreachable from that one location — even though it runs flawlessly for the rest of the world. A single-location monitor cannot distinguish "the site is down" from "my path to the site is down".
The solution is confirmation from multiple, independent locations. When one node reports a failure, the outage is only treated as real once further locations confirm it independently. If only one of three nodes sees a problem, that is a strong signal of a local network fault — and no reason to wake the entire on-call team. Only when several vantage points agree does a suspicion become a confirmed outage. This quorum principle is the most effective lever against path-induced false alarms.
Network faults are frequently one-sided: they affect a route, a peering or a resolver, not the service itself. Alerting from a single vantage point regularly mistakes your own connectivity for the state of the target site. Multi-node confirmation cleanly separates the two.
XICTRAQ starts exactly here. Reachability checks run on three independent nodes in separate data centres — DE-1, FI and DE-2 — two in Germany, one in Finland. An outage becomes an alert only when it is confirmed across nodes. If a single location drops out due to a local routing anomaly, the inbox stays quiet while the diagnosis records in the background which node saw what. The same logic also carries more complex synthetic browser checks, where a full click path rather than just an HTTP code is measured.
Remedy 3: hysteresis with separate thresholds
The third remedy tackles the "threshold at the edge" cause head-on. As long as a single threshold decides whether a state is up or down, any metric that hovers exactly on that line will inevitably flutter. Hysteresis solves this by introducing two separate thresholds: a higher one at which an outage is declared, and a lower one at which the all-clear is given again. Between them lies a dead band in which the state does not change — a buffer that absorbs the jitter. The principle is familiar from a thermostat: it does not switch on at exactly 21 degrees and off again at 21.0001, but works with a deliberate gap.
Nagios's own flap detection is, at its core, such a hysteresis. A host or service is deemed to be flapping "when its percent state change first exceeds a high flapping threshold", and only calm again "when its percent state goes below a low flapping threshold" (Nagios Core Documentation). The defaults make the gap concrete: the lower threshold sits at 5 percent, the upper at 20 percent state change (Nagios Core Documentation). Between 5 and 20 percent, nothing happens by design — this corridor is why a service once recognised as stable does not immediately count as unstable again on the next small fluctuation.
| Property | Single threshold | Hysteresis (separate thresholds) |
|---|---|---|
| Trigger logic | One boundary for up and down | High boundary for outage, low for recovery |
| Behaviour at the edge | Oscillates on every fluctuation | Dead band absorbs small excursions |
| Message volume | High, in time with the noise | Low, only on genuine transitions |
| Nagios default | not provided | 5 % recovery / 20 % outage (Nagios Core Documentation) |
| Risk | Permanent flutter at the edge | Marginally later but stable recovery |
Burn rate instead of a single sample: how SRE practice filters noise
The three remedies condense into an elegant pattern that Google SRE practice calls multi-window multi-burn-rate alerting. Instead of reacting to a single sample, the alert reacts to the burn rate — how quickly an error budget is consumed — and does so across two time windows at once. A long window detects the underlying problem, a short one checks whether it is still ongoing right now. The alert fires only when both windows exceed their threshold. The pattern thus combines confirmation over time (remedy 1) with built-in damping (remedy 3).
SRE practice gives concrete values for a 99.9 percent target. A page fires at a burn rate of 14.4 over a 1-hour window once 2 percent of the 30-day error budget is consumed; a second tier at burn rate 6 over 6 hours and 5 percent of the budget; a ticket at burn rate 1 over 3 days and 10 percent of the budget (Google SRE Workbook). As a rule of thumb: "A good guideline is to make the short window 1/12 the duration of the long window" (Google SRE Workbook). For the 1-hour window that means a short window of five minutes.
The dual-window approach reduces false positives because both conditions must hold at once, and it improves recovery: the alert stops firing five minutes after the problem ends, rather than an hour later (Google SRE Workbook). A brief spike that resolves on its own never exceeds the threshold in the long window at all — the transient noise falls out before it becomes a message.
Aligning your alerting to SLOs rather than to raw single values yields noise resistance almost as a side effect. The mechanics behind error budgets and burn rate — including how quickly a budget realistically burns down — are covered in detail in the article on error budgets and burn rate in SLO monitoring.
Deduplicate alerts: one incident, one message
Even with confirmed outages, one source of noise remains: repetition. One and the same incident must not report again on every check cycle. Deduplication groups related events into a single incident with one lifecycle — open when the outage is confirmed, closed when the recovery is likewise confirmed. Between those two points, exactly one message flows, not a steady stream. Repeated signals of the same incident are suppressed within a time window instead of flooding the inbox.
Each incident gets a deterministic key from target and error type. Repeats of the same key within a window are consolidated rather than reported individually.
When a service demonstrably recovers, the incident closes automatically and sends exactly one recovery message — no stale ticket that must be acknowledged by hand.
Instead of twenty individual emails, the team receives one consolidated message per incident over the chosen channel — from inbox to chat webhook.
Deduplication is especially valuable for periodic jobs, where a single stuck run would otherwise trigger a cascade. How to monitor such background processes cleanly in the first place, without alerting on every cycle, is shown in the guide to heartbeat monitoring for cron jobs. And because scheduled maintenance is a common source of flapping, the article on maintenance windows and false alerts is worth a look.
How XICTRAQ brings the three remedies together
The three remedies unfold their effect only in combination, and that is exactly how XICTRAQ implements them. Reachability checks run across nodes at three locations (DE-1, FI, DE-2); an outage becomes an alert only when it is confirmed across nodes — that is remedy 2 as default behaviour. Consecutive failures gate the state change over time (remedy 1), and incident deduplication ties everything into one lifecycle rather than reporting again on every wobble. A potential twenty-message storm thus becomes a single, actionable notification.
The operational rule of thumb is simple: an alert that is neither confirmed across nodes nor deduplicated is one alert too many. Putting those two filters consistently up front halves the message volume before thresholds are even discussed.
— Field experience
The result becomes visible on the status page and in reporting: a confirmed incident appears as one entry with a clear beginning and end, not as a fluttering chain. Anyone who wants to compare the tiers and the scope of multi-location confirmation will find the details in the pricing overview. Fleeting network anomalies and their detection are, incidentally, also the theme of adjacent articles on post-quantum TLS in monitoring and on monitoring encrypted mail delivery with MTA-STS, which show how broad the field of confirmation-worthy signals has become.
This article draws on: Nagios Core Documentation (Detection and Handling of State Flapping — 21-result window, at most 20 state changes, 50 percent higher weighting of the newest change, default thresholds of 5 percent and 20 percent, start/stop rule as hysteresis), Google SRE Book (Monitoring Distributed Systems / Effective Alerting — "what's broken, and why?", four golden signals, "every page should be actionable", "good signal and very low noise", the warning against ignoring real pages) and Google SRE Workbook (Alerting on SLOs — multi-window multi-burn-rate with burn rate 14.4 / 2 percent over the 1-hour window, 6 / 5 percent over the 6-hour window, 1 / 10 percent over the 3-day window, the short window as 1/12 of the long one, the dual-window condition for fewer false positives and faster recovery).
Frequently asked questions about flapping alerts
What exactly is a flapping alert?
How many consecutive failures should I require?
What does hysteresis mean in monitoring?
Why is a single check location not enough?
Does SLO-based alerting help against flapping?
How does XICTRAQ distinguish a real outage from noise?
A calm inbox, trust in every alert
Flapping is not a law of nature but the result of two fixable decisions: a threshold at the edge and a detector that turns a single sample into a diagnosis. Requiring consecutive failures, confirming from multiple locations and applying hysteresis with separate thresholds turns the signal-to-noise ratio around — towards fewer, actionable messages. That is exactly what monitoring aims at when it confirms across nodes and deduplicates rather than alerting on every wobble. The result is an inbox where every alert means something again. To see this on live monitors, the interactive demo shows a calm alert timeline without a login.