Deploy as Outage Cause: Roll Back in Minutes

The most likely cause sits in your own house

Google SRE published a figure from its own operations that has been quoted unchanged for years: roughly 70 percent of outages are due to changes in a live system (Google SRE). The second volume confirms it in the context of the error budget and calls changes a major source of instability (Google SRE Workbook). For an operation that releases weekly this means: the probability that the next incident starts on a Tuesday morning rather than in the small hours of Sunday is considerably higher than the security debate suggests.

The triggers are more specific still. An analysis of thousands of postmortems over seven years names the binary push as the most frequent trigger at 37 percent, followed by the configuration push at 31 percent (Google SRE Workbook). Together that is a good two thirds; changed user behaviour follows far behind at 9 percent. Among the root cause categories, the flaw in the development process ranks second at 20.23 percent and poor rollout planning fourth at 6.74 percent (Google SRE Workbook). The chapter on progressive rollouts likewise records that most incidents are triggered by binary or configuration pushes (Google SRE Workbook).

What sits behind those categories in practice boils down to five recurring patterns:

• One artefact with too many changes - three weeks of work in a single step. If something breaks, eighteen changes are under suspicion at once.
• A configuration that is only correct in the test system - different cache lifetimes, different headers, a different endpoint in the checkout.
• A dependency that changed quietly - an interface, a consent banner, a script from an external source.
• A deploy with no way back - the database schema was changed, the previous version no longer runs on it, a rollback would require a restore.
• A window with no ownership - released at 5 pm on Friday, and from 5:30 pm nobody is reachable.

All five have one thing in common: they can be planned. The attacker picks the moment; your own release does not - it is scheduled. That makes the deploy the only major cause of outages you can write into a calendar. Exactly that is where the discipline described in this article comes from.

What an outage costs

The cost side is well documented. In the annual survey behind the Uptime Institute outage report for 2026, 57 percent of respondents say their most recent major outage cost more than 100,000 US dollars; for the second consecutive year, one in five names a figure above one million US dollars (Uptime Institute). Roughly one in ten reports severe or serious consequences (Uptime Institute). For a mid-sized website the absolute numbers are smaller, but the ratio holds: the damage grows with duration, not with the severity of the original defect.

More remarkable than the amounts is the attribution of cause. For 2026, failure to follow prescribed procedures remains the leading driver of human-caused outages, alongside inconsistent or unclear processes and errors during installation and commissioning (Uptime Institute). That is not a tooling problem. A procedure nobody follows works like no procedure - and a procedure that exists only in a document is not followed during an incident, it is recalled.

The change window: when releases are allowed

A change window is a written agreement about when a release may happen and when it may not. It costs nothing and works immediately because it rules out the three most expensive circumstances: releasing without standby, releasing at peak load, releasing without witnesses. Five entries are enough.

1. A fixed slot - Tuesday and Thursday, 09:00 to 11:00, for example. Outside it, only fixes for an ongoing incident go out.
2. Named standby - one person executes, a second watches and measures. Both are reachable for the full duration of the window.
3. A baseline captured beforehand - measured values and screenshots of the key pages, so the later comparison has something to stand on.
4. An abort condition - written down in advance: which deviation triggers the rollback. Not negotiated during the window.
5. A slot for the post-deploy check - the release does not end with the upload, it ends with the measurement result.

The fourth point is where windows fail. As long as the abort condition is not fixed in advance, it gets negotiated during the event - by people who have just released something they believe in. A rule such as "more than two percent error rate over five minutes" or "one browser flow fails twice in a row" is uncomfortable, but it decides without discussion and without regard to person.

The baseline: measure before anything changes

Without a baseline every post-deploy check is a gut feeling. "The site feels slow" is not a statement; "the largest content area now loads in 3.4 seconds instead of 1.9" is. The baseline is therefore nothing grand: a fixed list of addresses, one set of values and one screenshot per address, captured immediately before the release. Ten minutes of work that save an hour of argument during an incident.

Three values are enough to start: status code, response time and the load threshold of the largest content area. A good measurement there is a load time of 2.5 seconds or less (web.dev). If you collect the web vitals continuously anyway, you already have the baseline and do not need to produce one. Add a screenshot per page type - home, category, service or product, form, checkout - so that change detection has something to compare against after the deploy. The list stays the same for months; only that way do you get a series instead of a snapshot.

The post-deploy check: five questions in ten minutes

The post-deploy check answers five questions, in this order: Does the server respond? Does the page look like it did before? Do the paths that earn money still work? Is the site still as fast as before? And: is it still findable? Each question has its own measurement point, and none of them is a human with a browser - that person does not measure, that person looks.

The fourth and fifth points are forgotten most often because they do not produce an outage but a delayed loss. Nobody reports a page that is one second slower after the deploy; a page accidentally set to noindex only becomes obvious when search traffic falls away. How such a deploy accident with noindex unfolds is a subject of its own. The same applies to stale content from a caching layer that keeps serving the old version for minutes after the release, and to consent tags that drift after a deploy.

Release in stages instead of all at once

The countermeasure Google SRE recommends has three parts: progressive rollouts, rapid and accurate problem detection, and safe rollback of changes. That combination effectively reduces the number of users and operations exposed to a bad change (Google SRE). The reasoning behind it is not technical but arithmetic: release a faulty version to five percent of traffic first and, in the worst case, you get five percent of the damage.

For a website on a single server the staging is coarser but not impossible: one directory per version and a reference that gets switched; a dry run on a second address; routing a share of the traffic through the server configuration. The effort is spent once, the benefit repeats with every release.

The rollback: practised, not improvised

A rollback that exists only on paper is not a rollback. It gets practised once a quarter without cause, preferably in the regular change window: release a version, trigger the rollback, measure the time, note the result. After doing that three times you know your own number - and it is usually well above the estimate.

#!/usr/bin/env bash
# Roll back to an existing version: switch the reference, flush caches, re-measure
set -euo pipefail

TARGET="/var/www/versions/$1"
test -d "$TARGET" || { echo "version $1 missing"; exit 1; }

ln -sfn "$TARGET" /var/www/current
systemctl reload nginx
redis-cli -n 2 FLUSHDB >/dev/null

for path in / /checkout/ /contact/; do
  code=$(curl -sS -o /dev/null -w '%{http_code}' "https://www.example.invalid${path}")
  printf '%s %s\n' "$path" "$code"
done

Three things determine the duration: whether the reference to the version can be switched instead of rebuilt, whether the caching layers are flushed in one step, and whether the database schema still serves the previous version. The third point is the most expensive. A column added in the new version does not obstruct the rollback; a column that was renamed or removed makes it impossible. The rule against that is simple: add first, switch over second, and clean up only one release later.

During an incident the question is never "what is the root cause?" but "how do I get back to the state of ten minutes ago within five minutes?". The root cause is settled by the postmortem the next day.

— XICTRAQ

What a rollback does not heal

A rollback takes back the change, not its consequences. That distinction is missing from most contingency plans and costs more time after the event itself - usually precisely when everyone believes it is over.

• Written data stays written - orders calculated with the wrong tax rate for forty minutes are still wrong after the rollback.
• Caches hold the broken version - services in front of the application, visitors' browsers and provider caches keep serving the faulty page until it is deliberately flushed or expires.
• Search engines saw the interim state - an accidental noindex takes effect as soon as it is fetched and does not vanish the moment it is withdrawn.
• Consent and measurement tags keep running - a script loaded without consent in the broken version collected data during that time.
• Accessibility stays broken - a removed label or a lost focus ring raises no alarm; for that you need continuous checking against the WCAG criteria.

Metrics that show the progress

Two metrics are enough to tell whether your release practice is improving. The first is the ratio of deployments that require immediate intervention following a deployment - as a rule a rollback or a quick fix (DORA). The second is the time it takes to recover from a deployment that fails and requires immediate intervention (DORA). Neither says anything about code quality and everything about control of the process.

Both are measurable without extra tooling as soon as the change window is recorded: number of releases, number of rollbacks, duration per rollback. After ten windows that gives a series, after thirty a statement. The only thing that matters is that the second metric is counted from the measurement point and not from the moment of realisation: the clock starts with the release, not with the customer's phone call.

Alerting paths during the window

The alert situation changes during the window. A team that normally keeps its notification paths on calm thresholds needs a sharper setting for two hours: shorter check intervals, lower thresholds, a second recipient. Afterwards it gets reset, otherwise the team gets used to alerts that mean nothing - and misses the one that means something.

The reverse direction is part of the alerting path. If you maintain a status page or a regular report, something should be posted there during the window, and before the first enquiry arrives. If an outage does happen, two sentences on the status page are the difference between an operator who knows and one who noticed nothing. For background processing that stalls after a deploy, only a heartbeat signal reports the standstill at all, and for the far end of an interface you need a check of the response structure, not just of the status code.

The postmortem after the event

A rollback deserves a postmortem - not as a formality, but because it proves that something made it through every check that should not have. Google SRE names on-call engineer intervention, expressly including a release rollback, as one of the usual triggers for a postmortem (Google SRE). Five questions are enough for the record:

1. What was the trigger? - the specific change, not "the deploy".
2. Why did the pre-release check not see it? - missing test case, missing configuration, missing data.
3. How long did detection take? - from the release to the first alert.
4. How long did the rollback take? - from the decision to the measured normal state.
5. What delayed the decision? - usually a missing or unclear abort condition.

Google SRE recommends defining postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary (Google SRE). The same sentence applies to the abort condition in the window. Both are decisions you make calmly or not at all.

A plan for the next four weeks

Getting started needs no platform, just four weeks and one person who writes it down. Each week ends with a result you can show.

1. Week 1: set the window. Two fixed slots, named standby, a written abort condition. One page of text is enough.
2. Week 2: create the baseline. Fixed page list, availability check, screenshots, first values for the largest content area.
3. Week 3: automate the post-deploy check. Record browser flows for login, form and checkout, trigger the run after the deploy, set thresholds.
4. Week 4: practise the rollback. Roll back once without cause, measure the duration, note the result, close the gaps you found.

After those four weeks the deploy has not become safer, it has become observable - and that is the actual progress. Change detection reports what visibly changed; the availability check reports whether the server responds at all; the recorded flows report whether the paths that earn money still work. The most likely cause of outages is then under observation, at a time the operation picked itself.