Health Check Endpoints: Shallow Checks Are Not Enough

What a shallow health check actually proves

The shallow check is quickly described: the web server accepts the request, the application framework resolves the route, a handler returns a fixed response. That proves exactly one thing - the process is alive and speaks HTTP. For the restart question that is the right answer. A restart fixes stuck threads, lost file descriptors and memory leaks inside the process. It fixes nothing that is broken outside it. Which is precisely why the shallow check must measure nothing else: every extra dependency in that probe turns a foreign fault into a local restart.

Trouble starts where the same answer is used for a different question. A load balancer does not ask whether the process is alive, it asks whether it may send customer traffic to this target. A process can run and still answer every business request with a server error because the database connection pool is empty. Answering both questions with the same endpoint means picking one of the two answers, usually by accident. What the probe interval contributes to detection time is covered in the article on choosing uptime check intervals and the resulting detection time.

• Exhausted connection pool: the database answers, but every new request waits for a free connection and runs into the timeout.
• Half-applied migration: the release renamed tables, the process runs, the queries hit nothing.
• Cold cache: after a restart every request goes all the way to disk and response time multiplies.
• Clogged queue: jobs are accepted but not processed - visible only hours later.
• Expired certificate on the far side: the application itself is healthy, the call to the external interface fails on the chain.

How much time an endpoint actually has can be read from the open web. Only 44 percent of mobile websites reach a good server response time, on desktop it is 55 percent (Web Almanac); the threshold for good sits at 0.8 seconds, and beyond 1.8 seconds it counts as poor (Web Almanac). A health endpoint belongs well below that threshold, because it does not run once but continuously. How to measure such times cleanly is described in the article on response time percentiles and the limits of the average.

Liveness, readiness and startup: three questions, three answers

Separating the probe types is not a matter of taste but a consequence of what the system does with the answer. The liveness probe answers: should this process be restarted? The readiness probe answers: should this target receive traffic? The startup probe answers: is the boot sequence complete? Three questions, three reactions, three entirely different demands on the check.

The most expensive merge is a liveness probe that checks the database. When the database fails, the platform restarts every container in turn. The restart repairs nothing but costs the warm cache, the open connections and the boot sequence. Afterwards the application is slower than before, fails the probe again and is restarted again. The readiness probe may check the database: its reaction is removal from rotation, and that is reversible as soon as the dependency returns.

The default values show how differently the three are meant. A container counts as broken only after three consecutive failures (Kubernetes documentation) - together with the default interval, roughly 30 seconds pass before a restart. The first probe, by contrast, runs with no lead time, immediately after the container starts (Kubernetes documentation); without a startup probe it hits an application that cannot possibly be ready yet. And for liveness and startup probes a single success is mandatory (Kubernetes documentation): the return to a healthy state cannot be debounced, so the swing back cannot be damped.

Default values decide more than your own code

A health endpoint is rarely queried from a single place. In a typical setup the container platform, the load balancer, the upstream web server and external monitoring all query the same path - each layer with its own cadence, its own deadline and its own threshold. Those values rarely live in your own configuration; they live in the defaults of the respective documentation. They are the actual contract the endpoint has to honour, and in places they contradict each other sharply.

At the load balancer the contradiction is most visible. By default the check runs only every 30 seconds (AWS), and the endpoint has five seconds to answer (AWS). Two failed checks are enough to take a target out of traffic (AWS) - but it returns to rotation only after five consecutive successful checks (AWS). An endpoint that occasionally runs into the deadline because it also probes an external interface loses a target for minutes even though the application itself did not fail. Measuring reachability from the outside exposes that gap immediately, which is the job of uptime monitoring with response time measurement.

A second load balancer even contradicts itself. Created through the graphical portal, it probes every five seconds (Microsoft); created through a template, the API or the command line, the interval is 15 seconds (Microsoft). A threefold difference in cadence for an identical intent - and it surfaces only during an incident. On top of that comes a deadline of at most 30 seconds (Microsoft); what takes effect, however, is the lesser of the probe interval and those 30 seconds - by default five or 15 seconds. A hanging database call inside the health endpoint keeps that connection open for that long and ties up a worker thread.

At the web server layer the cadence is tight and the debouncing weak - wherever there is a cadence of its own at all. The active health_check directive sits in the ngx_http_upstream_hc_module module and belongs to the commercial nginx Plus subscription (nginx); there the check runs every five seconds by default (nginx), and a single failed check is enough to drop a server from the group (nginx). Free nginx has no such active check: what remains is the passive failure counting through max_fails and fail_timeout, which only counts failures of real customer traffic - no separate cadence hits the health endpoint at all. On the classic web server the interval is 30 seconds, a single failure disables the worker (hcfails, Apache HTTP Server), and the dynamic check is not enabled in the shipped configuration at all (Apache HTTP Server). All of these cases lead to the same conclusion: the endpoint has to be cheap enough to tolerate the tightest of those cadences without a second thought.

The status code is the signal - not the body

A common construction fault sits not in the check logic but in the response: the endpoint neatly collects every partial state, writes them into a JSON document containing the word fail, and ships the whole thing with 200. To a human the answer is unambiguous. To every machine in front of it, the answer is success. The established response format governs exactly this point: a passing health check is answered with a code in the 2xx to 3xx range (IETF), a failing one with 4xx or 5xx (IETF). A 200 carrying fail in the body is formally wrong and is read as healthy by every upstream layer.

The third level is the interesting one. The warning level is deliberately delivered with a success code (IETF) - so the status code alone does not separate healthy from degraded; the body is needed for that. Hence the division of labour: the status code steers the machine, the body informs humans. Common application frameworks already model this; a service that is down maps to 503 by default, while everything else - including an unknown state - comes out as 200 (Spring Boot). Anyone introducing custom states has to maintain that mapping, otherwise a new state silently falls back to healthy.

HTTP/1.1 503 Service Unavailable
Content-Type: application/health+json
Cache-Control: max-age=3600

{
  "status": "fail",
  "version": "1",
  "serviceId": "checkout-api",
  "description": "order intake",
  "checks": {
    "postgres:responseTime": [
      { "componentType": "datastore", "status": "fail",
        "observedValue": 812, "observedUnit": "ms",
        "time": "2026-09-08T09:14:02Z",
        "output": "pool exhausted, 0 of 20 connections free" }
    ],
    "queue:depth": [
      { "componentType": "system", "status": "warn",
        "observedValue": 14203, "observedUnit": "jobs",
        "time": "2026-09-08T09:14:02Z" }
    ],
    "pdf-service:connectivity": [
      { "componentType": "component", "status": "warn",
        "observedValue": "degraded", "time": "2026-09-08T09:13:47Z",
        "output": "soft dependency, output is delayed" }
    ]
  }
}

How narrow the window for the status code is depends on the layer doing the checking, and the spread is considerable. The container platform treats every code from 200 to 399 as success (Kubernetes documentation) - which means a redirect to a login page counts as a healthy service. In the web server's active check every 2xx and 3xx code also passes as long as no explicit rule is set (nginx Plus). A load balancer, by contrast, can be strict enough that only code 200 counts as healthy: a 204 without a body or a 302 to the status page is already a failure there (Microsoft). And the success list can only be configured between 200 and 499 (AWS) - a 5xx cannot be registered as healthy there. Which codes actually show up in day-to-day operations is covered in the article on monitoring the 5xx server error rate.

When the deep check causes the outage in the first place

Deep checks have an awkward property: they are measuring instrument and load at the same time. Every call to the endpoint produces a database query, a connection to the queue, a call to the external interface. When the system is under pressure, that load lands on top exactly when the least headroom is left. If several layers each retry three times independently, a single click turns into an avalanche of 64 attempts on the database (Google SRE Book).

only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry

— Site Reliability Engineering, chapter 22: Addressing Cascading Failures

The recommendation behind it is to cap retries: at most 60 retries per minute and process, and beyond that budget no retry at all (Google SRE Book). If more than half of the instances fail the health check, a widely used proxy discards the result and sends traffic to all of them again - the default value of that emergency brake is 50 percent (Envoy). An overly strict check therefore disables itself, and it does so quietly: the console keeps showing instances as unhealthy while traffic has long since returned everywhere. How to tell such state swings apart from real outages is described in the article on taming flapping alerts across multiple locations.

If a service is already crash-looping, the same handbook suggests letting only one percent of traffic through at first (Google SRE Book) - only then can a cache fill and a connection pool build up. A health endpoint that measures full depth on every call prevents exactly that recovery: it reports unhealthy until the system is warm, and stops it from getting warm because it receives no traffic. Why foreign dependencies deserve particular attention here is covered in the article on cloud outages and monitoring third-party dependencies.

Separating hard from soft dependencies

The distinction that makes the difference is not technical but functional: without which dependency can this service not fulfil its purpose? For order intake the database is hard - without it there is no order. The service that renders the invoice as a PDF is soft: the order is accepted and the rendering goes into the queue. It changes with the functional scope, and it is the only defensible reason why a partial state may flip readiness or may not.

The absence of that separation has a measurable price. An availability target of 99.9 percent allows 43.2 minutes of downtime per month (Google SRE Book). Every unnecessary removal from rotation, triggered by a soft dependency, comes out of the same budget. How to spend that budget deliberately instead of burning it unnoticed is described in the article on error budgets and burn rate as an alerting basis.

Time budget, cache and a cadence of its own

A deep check without a time budget is not a check but a bet. The tightest limit in the setup sets the frame: the container platform grants a probe exactly one second by default (Kubernetes documentation), the load balancer five (AWS). A workable design gives each partial probe its own, distinctly smaller deadline, runs them concurrently and aborts the overall check as soon as the budget is spent. A partial probe that runs into its deadline then returns fail for that part - rather than an answer that fails to arrive at all.

The second lever is cadence. The deep probe does not have to run on every call. A background run measures the dependencies at its own rhythm and stores the result together with a timestamp; the endpoint then only reads that state and answers in microseconds. The response format explicitly anticipates this decoupling and recommends assigning a freshness lifetime to the health response so callers do not query the endpoint unthrottled (IETF).

The last line is the actual finding of this short series: the shallow path answers in four milliseconds, the deep one in a good two hundred - and the response carries no freshness lifetime, so every caller regenerates it at full cadence. Divergences like that between two paths of the same service can be cross-checked continuously from the outside once both endpoints exist as separate probes in API monitoring, each with its own expectations for code, runtime and content.

What belongs in the response and what does not

A health endpoint is an operations interface, not a system inventory. Anything an attacker could use to map attack paths does not belong in it: versions, internal addresses, connection strings, stack traces. The line is easy to draw when the endpoint has two tiers - a public one carrying only the overall state and a timestamp, and an internal one, unreachable from the network, that lists the partial states.

livenessProbe:
  httpGet:
    path: /health/live      # own process only, no network calls
    port: 8080
  periodSeconds: 10         # default value, confirmed here on purpose
  timeoutSeconds: 1         # default value - the path has to stay below it
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health/ready     # hard dependencies, read from the background run
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
  successThreshold: 2       # allowed here only: debouncing the swing back

startupProbe:
  httpGet:
    path: /health/started   # migration done, configuration loaded, warm-up complete
    port: 8080
  periodSeconds: 5
  failureThreshold: 60      # five minutes of boot time, then restart

• Overall state as a status code: one level, three values - pass, warn, fail - mapped cleanly onto the code ranges.
• Timestamp per partial check: states how old the statement is and exposes a frozen background run.
• Hard or soft marker: makes visible in the body why a partial state flips the overall state or does not.
• No secrets and no versions: neither credentials nor library levels, neither internal names nor stack traces.

For the response to stay correct as the application changes, it needs a cross-check from the outside. Expectations on status code, runtime and individual fields in the body record what a healthy answer looks like - and fire when a field disappears or a state silently falls back to healthy. How to cut such expectations sensibly is described in the article on assertions and thresholds in API monitoring.

Checking from the outside: the caller's view

Every probe from inside your own network shares one blind spot: it does not measure the path the customer takes. The endpoint can answer cleanly on the inside while a redirect, a firewall or an expired certificate chain changes the response in front of it. That is why every health endpoint deserves a second measurement from the outside, from a network that has nothing to do with your own. Only the comparison of both measurements reliably separates an application fault from a delivery fault.

The second question is what happens with the finding. It becomes useful once state changes are bundled and separated by severity - a failed hard dependency wakes someone, a warning soft one goes into the daily report. That separation is the actual purpose of the status levels in the body; it pays off only when alert delivery across separate channels reflects it.

For customers and colleagues what counts in the end is not the probe but the statement. A state change that flows from measured values into a status page with reporting saves half the follow-up questions during an incident - and makes it traceable afterwards when which partial state flipped.

Health checks as an operational decision, not a formality

A health endpoint is not a by-product but an interface with contractual character: it tells four layers at once what to do with a process. The three decisions that matter are quickly named - which dependency is hard, how much time the probe gets, and how state maps onto status codes. That ageing components bring pitfalls of their own is covered in the article on components without updates in the web stack.

The effort for the rework is manageable, the return is not: separate paths, a background run with a timestamp and three carefully chosen defaults prevent the two most expensive failure patterns - the dead target left in rotation and the fleet-wide restart loop. How much monitoring that takes depends on the scope; the overview of plans and their scope makes the difference between a single probe and coverage across all endpoints tangible.

1. Create three paths: /health/live without dependencies, /health/ready with hard dependencies, /health/started for the boot sequence.
2. Record for each dependency whether it is hard or soft, in writing.
3. Set a time budget per partial probe and keep the sum below the tightest deadline of the checking layer.
4. Map states onto status codes and secure that mapping with a probe from the outside.
5. Move the deep probe into a background run and serve its result with a timestamp.
6. Monitor both paths externally, with separate thresholds and separate delivery paths.