Why the Average Response Time Deceives
The arithmetic mean collapses all measurements into a single number - and loses exactly the information that matters. In a typical response-time distribution, most requests cluster closely together, while a few extremely slow outliers form a long tail to the right. The mean is barely moved by those outliers as long as they are rare. The result: a dashboard reports 180 milliseconds average while the slowest percent of users waits more than a second. Anyone monitoring only the mean never sees that delay (Google SRE Book).
A concrete example makes it tangible. Assume 10,000 requests to an endpoint. The median (p50) is 140 milliseconds - half of all requests are faster, half slower. The average sits only slightly higher at 180 milliseconds and looks harmless. Yet the 95th percentile (p95) stands at 620 milliseconds, and the 99th percentile (p99) at 1,240 milliseconds - nearly nine times the median. In other words: every hundredth request takes longer than a second, while the average stays undisturbed at 180 milliseconds. The same distribution, two entirely different stories - depending on which metric you read.
| Metric | Value | What it says | For outliers |
|---|---|---|---|
| Average (mean) | 180 ms | arithmetic mean of all requests | blind |
| Median (p50) | 140 ms | the middle request | blind |
| p95 | 620 ms | 5% are slower | sensitive |
| p99 | 1,240 ms | 1% are slower | very sensitive |
That is why monitoring thresholds should not be pinned to the average but to a high percentile. A threshold on the mean only fires once things are broadly bad; a threshold on p95 or p99 reacts as soon as a relevant share of users suffers. How to define such thresholds as concrete check rules is covered in the article on thresholds and assertions in API monitoring.
The Google SRE Book explicitly recommends capturing response times as a distribution - specifically as counts bucketed by latency, that is histogram buckets, rather than raw averages (Google SRE Book). Only then does the difference between a generally slow average and a slow tail become visible at all. A mean simply cannot represent that difference.
Understanding Percentiles: p50, p95 and p99
A percentile is a simple but powerful idea. The p-th percentile is the value below which p percent of all measurements lie. A p95 of 620 milliseconds means: 95 percent of all requests were answered in 620 milliseconds or faster, and only the slowest 5 percent took longer. A p99 of 1,240 milliseconds accordingly means that 99 percent of requests were faster and only the slowest percent lay above it. The median is nothing other than the p50 - the exact middle of the distribution.
- p50 (median) - the typical request. Good for describing the normal experience of the majority, but blind to outliers.
- p95 - the threshold where it gets uncomfortable. A good compromise between stability and sensitivity; it captures noticeable delays without jumping at every extreme value.
- p99 - the slowest percent. Shows the real outliers that cause timeouts and abandonment. Sensitive, but precisely for that reason meaningful for critical paths.
- p99.9 - for high-load systems. In large distributed architectures, even a tenth of a percent still affects a great many users per day.
The decisive advantage: percentiles describe what a real share of users actually experiences. While the average is an abstract computed figure that no single user ever encounters, the p95 corresponds to a concrete statement - one in twenty waits at least this long. It is exactly this translatability into real user experience that makes percentiles the foundation of any serious response-time monitoring.
Median and mean are identical only for symmetric distributions. Response times are practically never symmetric but right-skewed - with a hard lower bound, since nothing is faster than zero milliseconds, and an open long tail upwards. That is why the average of response times almost always lies above the median, and the two together say more than either number alone.
Long-Tail Latency: When One Percent Decides Everything
Why is the slowest percent so important? Because it amplifies dramatically in distributed systems. The much-cited analysis The Tail at Scale by Dean and Barroso provides the classic worked example: a server typically responds in 10 milliseconds but has a 99th percentile of one second. If a user request is served by just one such server, only one in a hundred requests is slow (Dean and Barroso, The Tail at Scale).
But modern applications rarely serve a single user request with a single server. If the request must collect responses from 100 such servers in parallel, then 63 percent of all user requests already take longer than one second (Dean and Barroso, The Tail at Scale). A seemingly rare one-percent problem becomes the majority experience. And even if only one in 10,000 requests exceeds one second at the server level, a service with 2,000 participating servers sees nearly one in five user requests above the one-second mark (Dean and Barroso, The Tail at Scale).
The 99th percentile of one backend can easily become the median response of your frontend.
— Google SRE Book, Monitoring Distributed Systems
The same work documents the amplification on a real Google service: the 99th percentile for a single random sub-request to finish is 10 milliseconds. The 99th percentile for all sub-requests of a user request to finish, however, jumps to 140 milliseconds - and waiting for the slowest 5 percent of sub-requests accounts for half of that total 99th-percentile latency (Dean and Barroso, The Tail at Scale). The long tail is therefore not a fringe phenomenon but the dominant cost factor of response time.
In practice, many small sources add up to this tail: a slow database query, an overrunning garbage collector, a cache miss, a TLS handshake that takes longer than expected. Monitoring response time per endpoint is how you find the source - the article on reliably monitoring REST endpoints shows the approach. The choice of encryption plays in as well: anyone planning the move to hybrid post-quantum schemes in TLS monitoring should measure the latency impact of the larger handshake deliberately in p95 and p99 rather than letting it vanish in the average.
A single slow backend service propagates through every layer that waits on it. In an architecture with many parallel calls, the perceived speed is set not by the fast median but by the slowest participating building block. That is precisely why a high percentile - not the average - is the honest metric for your users' experience.
Core Web Vitals: Why Google Measures at the 75th Percentile
That percentiles are the right lens shows nowhere more clearly than in the Core Web Vitals - Google's metrics for the real user experience on the web. Google assesses a page not by the average of its load times but at the 75th percentile of all page loads, segmented across mobile and desktop devices (web.dev). This ensures that the majority of visits actually reach the target level - the value was deliberately chosen so that most users experience the intended performance (web.dev).
The three current metrics and their thresholds for good are: Largest Contentful Paint (LCP) under 2.5 seconds for loading, Cumulative Layout Shift (CLS) of 0.1 or less for visual stability (web.dev), and Interaction to Next Paint (INP) of 200 milliseconds or less for responsiveness (Chrome for Developers). All three are assessed at the 75th percentile of field data, not at the average - a deliberate methodological decision.
INP is the most recent addition: on 12 March 2024, Interaction to Next Paint replaced the earlier metric First Input Delay as a Core Web Vital (Chrome for Developers). The reason fits the percentile idea exactly - First Input Delay measured only the first interaction, whereas INP considers the latency of all interactions in a session and thereby surfaces the slow outliers over the course of a visit (Chrome for Developers).
Largest Contentful Paint measures the load time of the largest content element - assessed at the 75th percentile of field data (web.dev).
Cumulative Layout Shift measures unexpected layout jumps - visual stability at the 75th percentile (web.dev).
Interaction to Next Paint measures the response time to inputs across the entire session (Chrome for Developers).
How to improve these values in the field is covered by the articles on Core Web Vitals optimization and specifically on optimizing the demanding INP metric. Single-page applications add a wrinkle explained in the article on Core Web Vitals and soft navigations. The Chrome field data itself is consistently reported at the 75th percentile (Chrome for Developers) - further proof that plain averages have had their day for judging user experience.
Setting p95 and p99 as Thresholds for Uptime and API Monitoring
Percentiles turn into practical monitoring thresholds. The rule of thumb: set limits at a percentile that matches the criticality of the path. A normal marketing page often gets by with a p95 target; a payment or login endpoint, where every delay costs revenue, warrants monitoring the p99 as well. The median serves as a reference for the normal state but does not alert - it reacts too sluggishly to outliers to serve as an alarm.
| Path | Monitored percentile | Example threshold | Reaction |
|---|---|---|---|
| Static marketing page | p95 | under 800 ms | ticket on breach |
| API endpoint (read) | p95 and p99 | p95 under 500 ms, p99 under 1,000 ms | alert on p99 breach |
| Checkout / login | p99 | under 1,000 ms | immediate notification |
| Background job | p99.9 | project-dependent | observe trend |
Fixed thresholds are a good start but reach their limits once load fluctuates by time of day. A response time that stands out in the morning may be perfectly normal at the lunchtime peak and still breach the same fixed limit. Here an adaptive baseline helps, learning the expected percentile per hour of the week and alerting only on genuine deviations. How synthetic checks measure whole user paths instead of individual endpoints is shown in the article on browser monitoring with synthetic checks.
The p99 is sensitive - which is intended, but carries the risk of noise. A single slow request over a short time window can briefly drive up the p99. That is why p99 alarms should be averaged over a sufficiently large time window and a minimum number of requests, so that a single outlier does not immediately trigger an escalation.
Deriving Real Alert Rules and SLO Targets from Percentiles
Percentiles are the natural building block for latency SLOs. An availability SLO describes how often a service is reachable; a latency SLO describes how fast it responds - and is almost always expressed via a percentile. A typical target reads: 95 percent of all requests are answered in under 500 milliseconds, measured over 30 days. That turns a vague expectation into a verifiable commitment the team can be held to.
The same mechanics can be attached to these latency SLOs as to availability SLOs: an error budget for requests beyond the threshold and multi-stage alarms that distinguish by consumption rate. How to build burn-rate alarms and error budgets from that is deepened in the article on error budget and burn rate in SLO monitoring.
- Choose the percentile - p95 for normal paths, p99 for critical ones; keep the median as a reference only
- Set the target value - a concrete threshold per endpoint, derived from the observed distribution rather than from gut feeling
- Define the measurement window - compute the percentile over a rolling window of roughly 30 days to smooth out daily fluctuations
- Stagger the alarm - fast notification on heavy consumption, a quiet ticket on slow erosion
- Refine regularly - adjust the threshold to the real distribution as usage or infrastructure changes
Response time is, moreover, only one dimension of service quality. Email delivery, for instance, relies on entirely different, transport-related safeguards - the article on MTA-STS and TLS-RPT for email encryption shows how quality can be monitored continuously there too. Good monitoring thinks about response time, availability and security together rather than in separate silos.
Capturing Response Times from Multiple Nodes
Percentiles are only as reliable as the measurement behind them - and measuring from a single location distorts the picture. A slow network path between one probe and your server drives up the p95 without a single real user being affected; conversely, a regional disruption goes undetected. That is why XICTRAQ® measures response time from three distributed locations and reports the percentiles per endpoint - separating network effects from genuine delays.
The gap-free response-time history at the same time provides the data basis from which p95, p99 and their evolution over time can be computed. From this history come thresholds, alarms and latency SLOs - all pinned to a high percentile instead of the deceptive average. The uptime monitoring by XICTRAQ® combines this measurement with multi-stage alerting via email, Slack, Discord or Telegram.
Response time from three distributed nodes - network effects of individual paths can be separated from genuine delays.
p50, p95 and p99 for each monitored path instead of a single global average.
A gap-free time series as the basis for percentile computation and trend analysis across weeks and months.
Expected response time learned per hour of the week - alerting only on genuine deviation rather than every load spike.
Notification via email, Slack, Discord or Telegram, staggered by urgency and escalation level.
Set limits per endpoint freely - p95 for normal, p99 for critical paths.
The shift from average to percentile is not a cosmetic change but a different view of the same data - one that shows what users experience instead of what the statistic smooths away. Anyone measuring p95 and p99 spots problems before they turn into abandonment and support tickets, and can act while there is still room to.
This article draws on data from: Google SRE Book - Monitoring Distributed Systems (an average of 100 ms hides that 1% of requests can take 5 seconds; the recommendation to capture response times as a distribution or histogram rather than a mean; the 99th percentile of a backend can become the median of the frontend), Dean and Barroso - The Tail at Scale, Communications of the ACM (a server with a 10 ms median and a 99th percentile of 1 second; 63% of user requests over 1 second with 100 parallel servers; nearly one in five user requests with 2,000 servers and 1 in 10,000 slow requests; a real Google example with a 140 ms 99th-percentile total latency, of which the slowest 5% account for half), web.dev - Web Vitals (thresholds LCP under 2.5 s and CLS up to 0.1; assessment at the 75th percentile of page loads, segmented across mobile and desktop) and Chrome for Developers (INP threshold of 200 ms; INP replaced First Input Delay as a Core Web Vital on 12 March 2024; field values are reported at the 75th percentile).