Worker pool exhaustion: why latency tips before the 502

Four limits in a row: where a request waits

A PHP application behind nginx passes through several stations on the way to a response, and each has its own limit. nginx accepts the visitor's connection and hands the request to PHP-FPM via FastCGI. PHP-FPM assigns it to a worker from its pool. The worker runs the script and usually queries a database. As long as there is spare capacity everywhere, nobody notices any of this. As soon as one station is full, a queue forms in front of it, and the waiting time adds to the actual processing time.

• nginx waits for the response from PHP-FPM. The default for fastcgi_read_timeout is 60 seconds (nginx documentation). After that nginx gives up and returns an error page.
• The socket queue holds connections for which no worker is currently free. On Linux, listen.backlog defaults to 511 (PHP Manual).
• The worker pool limits through pm.max_children how many requests are served at the same time (PHP Manual). Every further request waits in the queue in front of it.
• The database limits the number of concurrent connections with max_connections, which is 151 by default in MariaDB (MariaDB documentation).

The queue is the least visible part of this chain. It is managed by the kernel, not by PHP. Linux silently caps the value from listen.backlog to the content of /proc/sys/net/core/somaxconn (Linux manual page listen). Since Linux 5.4 that file defaults to 4096, in older kernels to 128 (Linux manual page listen). On an older system the effective queue can therefore be much shorter than configured. When it is full, according to the manual page the client either receives a rejection with ECONNREFUSED, or the connection request is ignored so that a later attempt succeeds (Linux manual page listen). Visitors see neither directly, only waiting time or an error page from nginx.

Why response time tips in steps

A full pool does not produce a single error at first. It produces waiting time. As long as the queue has room, every new request lands there and is eventually served. How long that takes depends mainly on two figures: how many workers run in parallel and how long a request takes on average. The following worked example is based on the assumption of 20 workers and on the Linux default of 511 slots in the queue (PHP Manual). It is deliberately simplified and not a measurement.

The example shows the core of the problem. Waiting time does not grow with load alone but with the product of queue length and processing time. If the database slows down because it is under load itself, the time per request goes up, throughput drops and the queue grows even faster. The curve therefore does not rise evenly but in steps: every additional slowdown in the background multiplies the waiting time at the front. Pages served from a cache stay fast meanwhile and hide the bottleneck, because they never reach PHP. Why an upstream cache can hide an outage at the origin is covered in the article on origin monitoring behind a CDN.

A full worker pool does not announce itself with errors but with waiting time. Whoever counts only error codes sees the bottleneck last.

— XICTRAQ

The nginx deadline: when waiting turns into a 504

nginx does not wait forever. For FastCGI, the directive fastcgi_read_timeout applies with a default of 60 seconds (nginx documentation). The documentation describes the deadline precisely: it applies only between two successive read operations, not to the transmission of the whole response. If the FastCGI server sends nothing within that time, the connection is closed (nginx documentation).

For the queue this has an awkward consequence. The connection to PHP-FPM already exists as soon as the kernel has accepted it into the queue. nginx has sent the request and is waiting for the first byte. The time a request spends in the queue therefore counts fully against the deadline. In the worked example above, a processing time of 2.5 seconds per request is enough for the rear slots of the queue to miss the deadline. Establishing the connection has a second deadline of its own, fastcgi_connect_timeout, also with a default of 60 seconds (nginx documentation). It applies when the connection is never established, for example because the kernel ignores connection attempts while the queue is full.

# Example: set the FastCGI deadlines explicitly, here with the default values
location ~ \.php$ {
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_pass unix:/run/php/www.sock;
    fastcgi_connect_timeout 60s;
    fastcgi_read_timeout 60s;
}

Raising the deadline only moves the 504 further back. Visitors then wait longer for the same error page, and the queue stays full. It is more useful to know the deadline deliberately, keep processing time well below it and measure the waiting time in front of it. An explicitly set deadline also has a practical benefit: anyone reading the configuration later sees the value instead of having to look it up in the documentation.

The worker keeps going although nobody is waiting

When nginx gives up, the work in PHP is not over. The worker initially notices nothing of the abort and keeps computing until the script finishes or a limit of its own applies. Exactly these workers are then missing for the next requests in the queue. PHP-FPM offers request_terminate_timeout for this: the timeout for serving a single request, after which the worker process is killed (PHP Manual). The manual recommends the option for cases where max_execution_time does not stop script execution, and lists 0 as the default, which means off (PHP Manual).

Two points matter here. First, this timeout only runs once a worker has taken over the request. The time spent in the queue before that does not count. Second, PHP-FPM kills the process hard. nginx then receives no valid response and returns a 502. A value just below the nginx deadline, around 55 seconds as an example value, makes sure that hanging scripts release their worker instead of keeping it busy while nginx has long since delivered an error page.

Why is max_execution_time not enough? Its default is 30 seconds (PHP Manual), but on non-Windows systems PHP uses it to measure only the time the script itself spends computing. According to the manual, system calls, stream operations and database queries are not included (PHP Manual). A script that waits almost a minute for an overloaded database has used hardly any time by that measure. The exception are builds with Zend max execution timers enabled, which measure elapsed time; the manual describes them with the directive.

; Example values for a pool with 20 workers, not a recommendation
[www]
pm = dynamic
pm.max_children = 20
listen.backlog = 511
request_terminate_timeout = 55s
request_slowlog_timeout = 5s
slowlog = /var/log/php-fpm/www-slow.log
pm.status_path = /fpm-status
pm.status_listen = /run/php/status.sock
ping.path = /ping

The slowlog is the counterpart to the hard timeout. With request_slowlog_timeout, PHP-FPM writes a backtrace of the running request to the slowlog file once the configured time has passed, without killing the worker; this option also defaults to 0, which means off (PHP Manual). The backtrace shows in which function the time goes, and with a full queue that is often a database query or a call to an external service. How to reduce the time to first byte in general is summarised in the guide to optimising server response time.

The second queue: database connections

Every worker that talks to the database needs a connection to do so. More workers therefore also mean more concurrent connections. MariaDB limits them through max_connections, which defaults to 151 (MariaDB documentation). This limit is shared by everything that accesses the same server: every PHP-FPM pool, every cron job, every background worker and every further application. As an assumption: three pools with 50 workers each already account for 150 possible connections before a single cron job runs.

According to its documentation, MariaDB always keeps one connection reserved for an account with the SUPER or CONNECTION ADMIN privilege (MariaDB documentation). That is the emergency access for administration. It only works if the application account does not hold this privilege; otherwise the application takes this last slot as well. Once the limit is reached, establishing a connection fails immediately. The application then returns an error page quickly, and exactly these fast errors later distort the measurement of response time.

Percentiles show saturation before the error rate

Google's book on Site Reliability Engineering describes the pattern in general terms. A web service with an average latency of 100 milliseconds at 1,000 requests per second can easily have 1 percent of requests that take 5 seconds (Google SRE Book). The average reveals little of this. The book calls rising latency an often leading indicator of saturation (Google SRE Book) and recommends measuring the 99th percentile response time over a small window, such as one minute, because it gives a very early signal (Google SRE Book).

For a full worker pool this means: the p95 or p99 rises first, the average follows sluggishly, the error rate comes last. On top of that there is a measurement error the book names explicitly. An error that is served quickly, for example because the database connection is missing, pulls the average response time down although the request failed (Google SRE Book). Response times should therefore be viewed separately for successful and failed requests. In addition, according to the book many systems degrade before they reach 100 percent utilisation (Google SRE Book). A pool that runs close to pm.max_children all the time has no buffer left for the next peak. How to read percentiles and use them for alerts is explained in the article on response time percentiles p95 and p99.

Telling 502, 503 and 504 apart

When waiting does turn into errors, the status code tells you where the chain broke. RFC 9110 defines the three codes clearly. A 502 means that a gateway or proxy received an invalid response from the server behind it (RFC 9110). A 503 indicates that the server cannot handle the request right now because of a temporary overload or maintenance (RFC 9110). A 504 means that the gateway did not receive a timely response from the server behind it (RFC 9110).

The order in which the codes appear is a finding in itself. If response time rises first and 504s follow, the queue is waiting too long. If 502s appear without a prior rise, either the full queue is refusing connections or workers are being killed. The nginx error log states the exact reason for each request. How to watch the error rate across all server errors is described in the article on the server error rate in 5xx monitoring. The meaning of individual codes can be checked with the HTTP status code lookup tool.

The FPM status page as an early warning

PHP-FPM ships with its own status page, which provides exactly the values missing from the log. It is switched on via pm.status_path and can output its values as text, HTML, JSON, OpenMetrics or XML. For the bottleneck, four fields matter most:

• listen queue: the number of requests currently waiting for a free process (PHP Manual). Anything above 0 means the pool is full at this moment.
• max listen queue: the highest queue level since the pool started. It captures peaks that fall between two polls.
• max children reached: shows whether the maximum number of processes has ever been reached; then it reads 1 or more, otherwise 0 (PHP Manual).
• slow requests: the number of requests that exceeded the configured request_slowlog_timeout (PHP Manual).

A status page that runs in the same pool as the application has an obvious problem: when the pool is full, it waits in the queue too. That is what pm.status_listen is for. The directive sets up a separate address for status requests and creates an invisible pool for it. The manual names exactly this case: the status page stays reachable even when the main pool is busy with long running requests (PHP Manual). The page reveals internals and therefore does not belong on the open internet. In the following example nginx only lets one fixed address through and additionally requires a secret token in the request header.

# Status page only for the monitoring check, address and token are examples
location = /fpm-status {
    allow 198.51.100.24;
    deny all;
    if ($http_x_status_token != "long-random-secret") { return 404; }
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
    fastcgi_pass unix:/run/php/status.sock;
}

The sample output above comes from a moment with a full pool and is shortened. A value above 0 for listen queue, zero idle processes and max listen queue at the level of listen queue len together show that the queue has already been filled to the brim. The PHP-FPM ping endpoint is no substitute. It answers with plain text and status code 200 (PHP Manual) and so only shows that PHP-FPM responds, not that workers are free. How to build a meaningful check endpoint is shown in the article on health check endpoints with deep checks.

For monitoring from outside, a small detour works well: a short script in its own small pool reads the status page through the status socket, asks the database for open connections and returns both as compact JSON. The same rules as above protect the endpoint, a fixed source address and a token. If one of the queries fails, the script reports that as a value of its own instead of filling in a zero, because a zero would look like an empty queue.

{
  "ok": true,
  "fpm": {
    "listen_queue": 0,
    "max_children_reached": 3,
    "active_processes": 12,
    "total_processes": 20
  },
  "db": {
    "threads_connected": 41,
    "max_connections": 151
  }
}

Assertions can be placed on this endpoint that report a queue before visitors feel it. Which assertions suit API responses is covered in more depth in the article on assertions and thresholds in API monitoring.

• status_code equals 200: the endpoint itself responds.
• json_path $.fpm.listen_queue less than 1: the queue is empty.
• json_path $.db.threads_connected less than 120: a self-chosen margin to the limit of 151 (MariaDB documentation).
• max children reached is a counter since the pool started. It belongs in the trend, not in a fixed assertion, because after it is first reached it stays above 0 until the next restart.

Measure the limit before visitors find it

Where exactly the chain breaks cannot be read from the documentation, only measured. A load test answers three questions: at how many concurrent requests does the queue start? How fast does the p95 rise from then on? And which error comes first, 502 or 504? Load is applied to your own systems only, ideally a staging environment with the same limits as production.

1. Read the effective limits: nginx -T for the deadlines, php-fpm -tt for the pool, SHOW VARIABLES for the database and sysctl net.core.somaxconn for the queue.
2. Set up a staging environment with the same values. A test against different limits measures a different system.
3. Raise the load in stages and hold each stage for a few minutes so the queue has time to fill.
4. During each stage, record p95, listen queue, max children reached and the open database connections.
5. Note which error appears first. A 504 points to excessive waiting time, a 502 to refused connections or killed workers.
6. Change only one limit at a time and measure again. Whoever adjusts pool, queue and database at once will not know afterwards which change had an effect.

Load peaks are often predictable. How a shop prepares for the holiday season is described in the article on Q4 traffic peaks. Anyone who wants to compare measurement series with the previous year later has to account for shifted sale days; the article on year-over-year Q4 reports with shifting sale days shows how. And anyone who keeps measurement series also has records that can help with evidence for cyber insurance.

How XICTRAQ makes the early signals visible

XICTRAQ measures from the outside, the way a visitor does. For a full worker pool that is the right perspective, because the waiting time in the queue does not show up in any PHP log, but it does show up in the response time a visitor experiences. The values from inside the server are added through the protected status endpoint.

How a baseline tells a normal Monday peak apart from a real bottleneck is described in the article on anomaly detection in website monitoring. What XICTRAQ does not do: it does not look inside the server itself. The values from the FPM status page only arrive if a protected endpoint exposes them, as described above.