On-Call Fails at the Roster, Not at the Tool
The typical entry into on-call looks like this: monitoring gets set up, alerts land on the phone of whoever knows the system best, and the rest of the team hopes for the best. That holds - as long as nothing happens. The moment the first night-time outage arrives, the gap shows: there is no backup, no documented response, no rule for when to escalate. The one person who knows everything becomes the bottleneck and the risk at the same time. If they drop out - holiday, illness, resignation - the entire on-call setup drops out with them.
The Google SRE team explicitly frames on-call not as a side activity but as plannable work with a fixed time budget. The rule there is that at least 50 percent of an SRE's time should go into engineering work and no more than 25 percent into on-call duty (Google SRE Book, Chapter 11). These numbers are not a comfort promise but a capacity calculation: anyone spending more than a quarter of their time on call has no time left to fix the causes that wake them at night. The rotation then becomes a self-reinforcing loop - more incidents, less time for stability, more incidents again.
A second point from the same source is just as central: three resources noticeably reduce on-call stress - clearly defined escalation paths, established incident-management procedures, and a blameless postmortem culture (Google SRE Book, Chapter 11). If one of them is missing, the person on duty carries not only the technical load but also the uncertainty about whether a mistake will be pinned on them personally. That uncertainty is exactly what turns a demanding night into a damaging one - and a solvable staffing question into attrition.
When on-call duty does not appear as its own line in the time budget, it becomes invisible. The Google SRE model reserves it explicitly: no more than 25 percent of working time for on-call, at least 50 percent for engineering work (Google SRE Book, Chapter 11). What is not planned is not bounded - and what is not bounded grows.
How Many People a Viable Rotation Needs
The first concrete question for every small team is: can the three of us manage? The answer from the SRE literature is uncomfortably clear. For a round-the-clock rotation at a single site, the Site Reliability Workbook names a minimum of nine people - eight as the base plus one person as a buffer for holidays, illness and turnover (Google SRE Workbook, Chapter 8). The SRE Book itself names a minimum of eight people for a single-site rotation with week-long shifts and one primary plus one secondary on-call role (Google SRE Book, Chapter 11). Both figures describe the same reality from two angles: round-the-clock coverage is not sustainable with a handful of people.
If the team is spread across two sites in different time zones, the need per site drops considerably: six people per site then suffice - five plus a buffer (Google SRE Workbook, Chapter 8). The reason is simple: a follow-the-sun model needs no night shift, because one site's night is the other site's working day. For most small and mid-sized German companies this is not a realistic option, which is why nine remains the honest benchmark.
| Model | Minimum size | Night shift needed | Source |
|---|---|---|---|
| Single site, round the clock | 9 people (8 + 1 buffer) | Yes | Google SRE Workbook, Chapter 8 |
| Single site, weekly primary and secondary shifts | 8 people | Yes | Google SRE Book, Chapter 11 |
| Multi-site, two time zones | 6 people per site (5 + 1) | No | Google SRE Workbook, Chapter 8 |
| Small team under 6 people | Only with a deliberately reduced coverage window | Depends on scope | Project experience |
What should small teams do when they cannot reach these numbers? The most honest path runs not through more discipline but through less ambition. A team of four can run a dependable on-call setup - just not round the clock for everything. The sensible move is to limit the coverage window deliberately, say to 07:00 to 23:00, and to let only a very small, clearly defined set of customer-facing outages wake anyone for the rest of the night. Everything else waits until morning. This decision belongs in writing and agreed with management - not made tacitly and then carried alone by whoever is on duty (project experience).
Both Google models explicitly budget one person above the arithmetic minimum: nine instead of eight for single-site, six instead of five per site for multi-site (Google SRE Workbook, Chapter 8). The buffer absorbs holidays, illness and resignations. Planning without it does not give you a rotation - it gives you a rotation in permanent exception mode, where every absence topples the plan.
Two Incidents per Shift as the Capacity Limit
The most important number in this article is an upper bound, not a target. The Google SRE team established a maximum of two incidents per 12-hour on-call shift (Google SRE Book, Chapter 11; Google SRE Workbook, Chapter 8). This limit is not a gut call but the result of a calculation: handling an incident completely - root-cause analysis, remediation and follow-up work such as writing a postmortem and fixing bugs - is estimated there to take roughly six hours (Google SRE Book, Chapter 11). Two incidents at six hours each fill a 12-hour shift exactly. A third simply does not fit - it inevitably gets handled superficially.
The counting unit matters here: an incident is one problem, regardless of how many alerts fire for it (Google SRE Workbook, Chapter 8). A database outage that triggers fifteen alerts across eight services is one incident - not fifteen. Counting alerts instead measures the loudness of the monitoring, not the load on the team. The reverse is also true, though: fifteen notifications for one incident are a problem in their own right, because the person on duty has to sort the noise before they can start working.
Distribution matters just as much. The SRE Book explicitly notes that the distribution of paging events over time should be very flat, with a likely median value of zero (Google SRE Book, Chapter 11). An average load of two incidents per shift sounds bearable, but in reality it often means six quiet shifts and one in which eight incidents escalate at once. The average conceals exactly the nights that wear people down. Anyone assessing the load should therefore look not at the mean but at the worst shift of the last eight weeks.
If more than two events occur regularly per on-call shift, the incidents cannot be investigated thoroughly, and the engineers are sufficiently overwhelmed that they no longer learn from these events.
— Google - Site Reliability Engineering, Chapter 11 (Being On-Call), paraphrased
When the mark of two incidents per 12-hour shift is exceeded persistently, the obvious reaction is to pull more people into the rotation. That is usually the wrong conclusion. The limit is a statement about the system: it alerts too often, too imprecisely - or it really is too unstable. More people only spread the load wider, they do not reduce it. How to cut the background noise at the root is covered in the article on configuring monitoring alerts properly.
Which Alert May Wake You at Night
The question of which incident is allowed to ring a phone at three in the morning decides whether the whole rotation holds. The Site Reliability Workbook states the principle in one sentence: every alert should be immediately actionable - there must be an action a human takes right after receiving the page that the system cannot take itself (Google SRE Workbook, Chapter 8). Most night-time alerts fail this test. A certificate expiring in 20 days requires no action at three in the morning. A checkout that stops accepting orders certainly does.
In practice a three-tier classification has proven useful, as the Workbook describes using one operator's example: P1 pages immediately, is SLO-impacting and immediately actionable; P2 goes out as an email for the next business day and covers matters without direct customer impact or with limited scope; P3 is purely informational and lives on a dashboard without notifying anyone (Google SRE Workbook, Chapter 8). What matters is not the naming but the discipline: every alert gets exactly one tier, and the P1 tier stays short. A P1 list that grows to twenty entries is no longer a classification - it is a wish list.
| Tier | Channel | Example | Wakes you at night |
|---|---|---|---|
| P1 - Page | Phone, push, pager | Checkout stops accepting orders; site unreachable from multiple locations | Yes |
| P2 - Ticket | Email for the next business day | Certificate expires in 20 days; response time over budget but no outage | No |
| P3 - Information | Dashboard, no delivery | Single check blip without confirmation from other locations | No |
The dividing line fits into one sentence: at night, wake someone for what affects customers right now and what a human can fix right now. Everything else waits. An expiring certificate is the textbook case for the second category - it is important, but the deadline is measured in weeks, not hours. The sensible approach is a staged lead time that surfaces early as a ticket and only escalates once the remaining validity genuinely becomes critical. How to set that up in detail is described in the article on SSL certificate monitoring.
- Does it affect customers right now? If not, it is not a page - no matter how urgent it feels
- Can a human fix it right now? If the action waits until morning anyway, the alert waits too
- Is the incident confirmed? A single blip from one probe is not enough - only confirmation from other locations makes it an outage, as the article on taming flapping alerts explains
- Is there a runbook? An alert without a documented response wakes someone who then has to improvise
- Is a maintenance window running? Planned work must not trigger a page; how to separate the two cleanly is covered in the article on maintenance windows and false alerts
The Runbook: The Answer to the Three-in-the-Morning Question
A runbook is the difference between an alert anyone on the team can work through and one that necessarily requires the single expert. The Site Reliability Workbook is unambiguous here: the playbook should be created together with the alert, not afterwards (Google SRE Workbook, Chapter 8). An alert without a runbook is an unfinished alert. The rule sounds strict, but it is the most effective lever against the situation where someone woken at three in the morning first has to research what their own system is trying to tell them.
The Workbook outlines runbook content concretely: high-level instructions for responding to the automated alert, an explanation of severity and user impact, debugging suggestions, and possible mitigation actions (Google SRE Workbook, Chapter 8). Translated into the daily reality of a small team, that yields six building blocks which should fit on a single screen:
A plain-language description of the alert and the condition that triggered it - not a repeat of the metric name, but its meaning.
Severity and concrete user impact: who currently cannot do what? This detail decides the pace of the response.
The two or three places where the cause typically becomes visible - dashboard, log, the provider's status page.
Mitigation actions that stop the effect before the cause is understood - rollback, rerouting, switching off a feature.
The clear condition for calling the next tier - by elapsed time, by scope, or by lack of progress.
The follow-up: postmortem, ticket, alert adjustment. The incident is over only once the cause has been addressed.
The runbook is also the tool that brings new people into the rotation. Google describes an onboarding list of roughly 24 focus areas a new team member should have practised before their first own on-call shift - from administering production jobs through draining traffic and rolling back software to understanding the service architecture (Google SRE Workbook, Chapter 8). For a small team the list is shorter, but the principle stays the same: nobody enters their first night shift without having practised the most common moves once in daylight. A rotation where half the participants can only forward things is not a rotation - it is a holding queue.
On-Call Duty in Germany Is Regulated by Employment Law
This is where the reach of the international SRE literature ends. The Google books describe a capacity and culture model; they say nothing about German employment law. In Germany, however, on-call duty is not a purely organisational matter but a regulated one. Anyone building a rotation should know the essentials before the roster is fixed. This section places the legal position in general terms and does not replace legal advice in an individual case.
The Court of Justice of the European Union has drawn the decisive line. In its judgment of 9 March 2021 in case C-580/19, a firefighter employed by the city of Offenbach am Main did not have to be at a place designated by his employer during standby, but had to be reachable and able to reach the city limits within 20 minutes in his emergency clothing and with the emergency vehicle provided (CJEU, C-580/19). The Court held that such standby time counts as working time in full only if the restrictions imposed significantly impair the worker's ability to organise their free time (CJEU, C-580/19). The Court thereby rejected any blanket classification: what matters is the response deadline and the average frequency of actual call-outs.
How far this can reach is shown by the earlier decision C-518/15 of 21 February 2018: there, a firefighter had to spend standby at home and respond to a call within eight minutes. That deadline significantly restricted the ability to pursue other activities - the Court classified the time as working time (CJEU, C-518/15). An important distinction applies to pay: the European Working Time Directive does not govern remuneration; that follows national law, the collective agreement or the employment contract (CJEU, C-518/15). Working time under the directive and paid time are therefore two separate questions.
| Case | Requirement | Classification | Criterion |
|---|---|---|---|
| CJEU C-518/15 (2018) | Standby at home, response within 8 minutes | Working time | Deadline significantly restricts use of free time |
| CJEU C-580/19 (2021) | Free choice of location, city limits within 20 minutes with vehicle | Working time only where impairment is significant | Response deadline and average call-out frequency |
| Typical IT on-call duty | Reachability, laptop, callback within 15-30 minutes | Case-by-case assessment on the same criteria | How strongly the requirements bind free time |
For IT on-call practice, one very concrete consequence follows: the response deadline is the lever. A deadline of five minutes, which the SRE Book names as a target for user-facing, highly time-critical services (Google SRE Book, Chapter 11), effectively ties the person to their device - shopping, sport or a cinema visit are out. A deadline of 30 minutes, named there for less time-sensitive systems (Google SRE Book, Chapter 11), leaves considerably more room. Setting the deadline tightly without need not only sharpens the burden; under the CJEU criteria it also pushes the standby closer towards working time. The deadline should therefore be derived from the availability target - not from caution.
Regardless of how standby is classified, the requirements of the German Working Time Act (Arbeitszeitgesetz, ArbZG) remain untouched. Section 5 ArbZG requires an uninterrupted rest period of at least eleven hours after the end of daily working time (ArbZG Section 5). Section 3 ArbZG caps daily working time at eight hours and permits an extension to ten hours only if the average stays at eight hours over six calendar months or 24 weeks (ArbZG Section 3). A night-time call-out that counts as working time interrupts the rest period - the next day's start shifts accordingly. That belongs in the roster, not in improvisation the morning after.
In practice this means: a call-out at two in the morning lasting an hour has consequences for the following morning. That consequence can be planned - a rotation that keeps the day after a night-time call-out free is not generosity but the orderly handling of a statutory requirement. Documenting the rotation also documents compliance with it; and evidence obligations have become routine in regulated contexts anyway, as the article on demonstrably vetting service providers under NIS2 shows.
Handover, Escalation and Follow-Up
Three processes hold a rotation together: the handover between two shifts, the escalation within a shift, and the follow-up afterwards. The Site Reliability Workbook recommends a shift length of no more than 12 hours and explicitly considers 24 hours or more without reprieve unsustainable - exhaustion increases the rate of mistakes (Google SRE Workbook, Chapter 8). That makes the handover a recurring, plannable event rather than an accident.
- Hand over state, not just events - the outgoing shift passes on open incidents, running workarounds and known wobblers, not merely a list of closed tickets
- One primary and one secondary on-call role - the secondary tier catches what the primary misses or cannot absorb; the SRE Book explicitly budgets the rotation with both roles (Google SRE Book, Chapter 11)
- Automatic escalation on time - if an alert is not acknowledged, it moves to the next tier without manual intervention; nobody should have to rely on it, but nobody should be able to sleep past it either
- A clear condition for calling upwards - at what scope, or after how long without progress, do you escalate? That threshold belongs in the runbook, not in the night
- Blameless postmortem - the follow-up addresses the cause, not the person; this is precisely the culture the SRE Book names as one of the three stress-reducing resources (Google SRE Book, Chapter 11)
- The alert itself goes under review - every incident raises the question of whether the triggering alert was actionable; if it was not, it gets adjusted or switched off
The path from incoming alert to resolved disruption deserves its own practised structure. How to separate roles, communication and escalation tiers cleanly is covered in the article on incident management and escalation. And how much headroom is left at all before an availability target breaks is made visible by looking at error budget and burn rate - which supplies the objective basis for deciding whether an incident is handled at night or in the morning. Both pieces interlock: the budget says how urgent it is; the rotation says who takes it.
Coupling the Rotation to Monitoring
A roster without a technical counterpart stays a document. The coupling to monitoring decides whether the rotation holds when it counts: the alert must reach the person on duty right now - not the one who was on duty three weeks ago. It must escalate when nobody responds. And it must distinguish a confirmed outage from a single probe's blip, because a false alarm at three in the morning costs the same night as a real incident. Why a single probe is not enough for that is explained in the article on multi-location monitoring.
Separate channels for page and ticket - urgent incidents over a channel that wakes you; everything else over one that waits.
If an alert stays unacknowledged, it moves automatically to the next on-call tier - without anyone having to chase it manually.
XICTRAQ checks from three distributed nodes; an outage counts as confirmed once several probes report it independently.
Planned work triggers no alert and does not weigh on the availability statistics.
Expiring certificates and domains announce themselves weeks ahead as a ticket - not as a night-time page.
The gap-free incident history shows whether the limit of two incidents per shift holds - and where the noise comes from.
The last point closes the loop back to the capacity limit. Without history, the question of whether the rotation is overloaded stays a matter of feeling - and feelings distinguish poorly between a hard week and a structural problem. With history it becomes a matter of arithmetic: count the incidents per shift over eight weeks, compare against the two, and the answer is there. Alerting with escalation tiers and separate channels supplies the data basis for that along the way.
Anyone wanting to protect the rotation against external triggers as well should keep an eye on the sources that produce unplanned nights. A freshly published vulnerability in a component you run is a classic cause; how to track such advisories in a structured way is described in the article on using EUVD and CVE vulnerability feeds. And the pricing overview shows from which tier escalation tiers and multi-location confirmation are available.
An On-Call Setup People Can Sustain for Years
Building a rotation is not a question of tooling but a series of decisions that belong in writing. How many people carry the plan - and which coverage window can that number honestly deliver? What response deadline applies, and is it set as generously as it can be? Which alerts may wake someone, and does each of them have a runbook? How does escalation work when nobody responds? And what happens on the day after a night-time call-out? Anyone who can answer these five questions in writing has an on-call setup. Anyone who cannot has a hope.
The most reliable touchstone remains the number from the SRE Book: no more than two incidents per 12-hour shift (Google SRE Book, Chapter 11). If it is exceeded over several weeks, that is not an invitation to compress the roster but the signal to make the system quieter - through more precise thresholds, through confirmation from several locations, through consistently switching off alerts that carry no action. A rotation that holds up against this limit can be run for years. One that ignores it tends to lose the people first and the availability second. The measurement basis for that comes from uptime monitoring with a gap-free history, and a look at the no-obligation demo shows the escalation logic in operation.
This article draws on: Google - Site Reliability Engineering, Chapter 11 (Being On-Call) - upper bound of two incidents per 12-hour shift, roughly six hours of effort per incident including follow-up, minimum of eight people for a single-site rotation with primary and secondary on-call, no more than 25 percent of time on call and at least 50 percent on engineering work, response times of 5 and 30 minutes depending on time criticality, flat distribution of paging events with a likely median of zero, three stress-reducing resources (escalation paths, incident-management procedures, blameless postmortems). Google - The Site Reliability Workbook, Chapter 8 (On-Call) - minimum of nine people for a single-site round-the-clock rotation (eight plus buffer) and six people per site in the multi-site model (five plus buffer), shift length of no more than 12 hours, target of at most two incidents per shift, one incident equals one problem regardless of alert count, P1/P2/P3 classification, the principle that every alert must be immediately actionable, playbook content and the point at which it is created, roughly 24 focus areas in onboarding. CJEU, judgment of 09/03/2021, Case C-580/19 (Stadt Offenbach am Main) - standby time counts as working time in full only where the impairment of free-time use is significant; the criteria are the response deadline and the average call-out frequency; facts involved a 20-minute deadline to the city limits. CJEU, judgment of 21/02/2018, Case C-518/15 (Matzak) - standby at home with an eight-minute response deadline classified as working time; remuneration is not governed by the Working Time Directive. German Working Time Act (Arbeitszeitgesetz, ArbZG), Federal Ministry of Justice - gesetze-im-internet.de - Section 3 (daily working time of eight hours, extension to ten hours where averaged out over six calendar months or 24 weeks) and Section 5 (uninterrupted rest period of at least eleven hours after the end of daily working time). Figures marked (project experience) come from our own monitoring projects. The employment-law section places the legal position in general terms and does not replace legal advice in an individual case.