Order heartbeat: get alerted when orders stop coming in

Why a silent checkout failure goes unnoticed for hours

Most disruptions in online retail are not complete outages but defects at exactly one point of the checkout. A plugin update changes the order of the scripts and the payment window no longer loads. An access key at the payment provider expires and the order remains in the status "open". A new shipping cost rule returns no price for certain postcodes and the order button stays grey. In all three cases the website is reachable and the cart fills up; only the last step fails.

A classic uptime check sees none of this, because it evaluates the status code and response time of an address. The conversion rate does not reveal the defect straight away either, because abandoned purchases are normal: averaged across 50 studies, the documented cart abandonment rate is 70.22 % (Baymard Institute). A few extra percentage points on one afternoon hardly show in a daily report. The outage is then noticed in the evening in the order list or the next morning through a customer email.

Google's site reliability handbook separates two questions for this: what is broken and why (Google SRE Book). Alerting should focus mainly on symptoms, and on causes only when they are definite and imminent (Google SRE Book). For a shop, the clearest symptom is simply that orders stop arriving. Whether the cause lies in the payment module, the shipping logic or the payment provider, the order heartbeat does not need to know in order to raise the alarm.

Your monitoring system should address two questions: what’s broken, and why?

— Google, Site Reliability Engineering, chapter 6 "Monitoring Distributed Systems"

In the fourth quarter every hour carries more weight. Around one in two people (52 percent) planned to use the reduced prices on Black Friday 2025, the vast majority of them online; 1,120 internet users aged 16 and over were surveyed (Bitkom). For the 2025 Christmas season the German retail federation expected online sales of 22.2 billion euros (HDE), and German retail as a whole generates 18.5 percent of its annual sales in November and December (HDE). How to prepare the website itself for this peak window is covered in the article on Q4 traffic peaks.

The principle: the shop reports what it has sold

A heartbeat turns monitoring around: instead of a checker fetching a page from outside, a process reports from inside at a fixed rhythm. If the report does not arrive, an alert follows. The article on heartbeat monitoring for cron jobs explains this dead man's switch in detail. With an order heartbeat, a small counting job checks in and sends a business figure along: the number of orders in the last time window, passed as the items parameter.

XICTRAQ® heartbeats evaluate this parameter when the setting "Job must produce output/items" is enabled on the heartbeat (field expect_output), optionally with a minimum count (min_items). This results in four states:

• Ping with items at or above the minimum: healthy run, an open alert is resolved automatically.
• Ping with items=0 or below the minimum: alert "Heartbeat without result" with the reported and the expected count, severity warning. The counting job ran, but the shop sold nothing or too little.
• No ping by the next due time plus grace period: alert "Heartbeat missed", severity critical. The counting job itself has failed, for instance because the server is down or the cron service is not running.
• Failure reported via /fail or exit_code other than 0: alert "Heartbeat failed", severity critical. The counting job could not count and says so explicitly.

The third state sets the order heartbeat apart from a database query that sends an email at zero orders: if that query's server goes down, it falls silent, and silence looks like a quiet day. The heartbeat expects the report and fires when it is missing. An open alert is updated on further runs instead of being created twice. Who receives it and through which route is defined by the account's alert channels and escalation levels.

The counting job: one query and one call

The counting job runs where the orders are: on the shop server or on a machine with read access to the shop database. It needs a read-only database user, a query and an HTTP call. Count orders that indicate a successful checkout, meaning neither cancelled nor flagged as tests. Counting only paid orders additionally reveals missing payment confirmations, but invoice and prepayment then need separate handling. Table and column names differ between shop systems; the example shows the pattern.

#!/usr/bin/env bash
# Counts the orders of the last N minutes and reports them to a heartbeat.
# Usage: count-orders.sh <heartbeat-slug> <window-minutes>
set -euo pipefail

HB="https://api.xictraq.com/api/heartbeat/$1"
WINDOW="${2:-60}"

# If the query fails, the job reports a failure right away instead of aborting silently.
trap 'curl -fsS -m 10 "$HB/fail?msg=query%20failed" >/dev/null' ERR

# created_at is stored in UTC, hence UTC_TIMESTAMP() rather than NOW().
N=$(mysql --defaults-extra-file=/etc/shop/read-only.cnf -N -B -e "
  SELECT COUNT(*) FROM orders
  WHERE created_at >= UTC_TIMESTAMP() - INTERVAL ${WINDOW} MINUTE
    AND status NOT IN ('cancelled', 'test')")

curl -fsS -m 10 "$HB/ping?items=${N}" >/dev/null

Two lines do the heavy lifting. The trap reports a failed query as a failure right away. Without it, set -e would make the script abort silently, and the alert would only come after the due time and grace period as "Heartbeat missed", with no hint at the cause. With it, "Heartbeat failed" and the reason appear in the alert at once, kept apart from the "no orders" case. A failed measurement is not a measurement. The last line is the actual ping; further parameters such as exit_code, duration_ms or a note via msg are described in the heartbeat documentation.

Before the heartbeat goes live, a manual cross-check is worthwhile. A ping with items=0 must report the idle run (the response text is in German), a ping with a normal count must not:

How many orders are normal? Calculating the threshold

The most important decision is the threshold, and it depends on your own order rate. Orders arrive randomly. The Poisson distribution suits such counts; it models the number of events occurring within a fixed time interval (NIST), and its parameter λ is the average number of events in that interval (NIST). The probability that not a single order arrives in an undisturbed window is therefore e to the power of minus λ. The table shows what this means for a counting job that runs 15 times during the day.

A rigid rule of "zero orders in an hour equals alert" therefore does not work for many shops. With an average of two orders per hour, roughly every seventh hour would mathematically pass without an order, with no defect at all (NIST), and the alert would fire around 60 times a month for no reason. The site reliability handbook describes the consequence: people who are paged too often skim or ignore alerts and then miss the real ones too (Google SRE Book). From five expected orders per window, the probability of a randomly empty window drops below one percent.

The model simplifies: real orders arrive in clusters, for example after a newsletter, and in between it is quieter than calculated. The values are therefore a lower bound for false alarms, not a promise. You read λ from your own data: count the orders of the last four to eight weeks per hour and weekday, calculate the mean and count how often an hour actually passed without an order. If that share deviates clearly, your own value applies.

Aligning counting window, rhythm and minimum count

Three settings determine how quickly and reliably the heartbeat fires: the length of the counting window, the rhythm of the counting job and the minimum count. Window and rhythm do not have to be equally long. A job that counts the last two hours every 30 minutes has a large λ and few false alarms, but only notices a complete outage once the entire window is empty. The detection time of a zero alert lies between one window length and one window length plus one interval.

With min_items, the heartbeat fires not only at zero but as soon as fewer than the minimum are reported. With ten expected orders and a minimum of three, chance alone mathematically triggers in about 0.28 % of runs (NIST), yet in a complete outage the alert comes earlier because the window does not have to be completely empty. Important: the minimum count is a fixed value per heartbeat. XICTRAQ® offers an expectation that adapts per hour of the week for the runtime of a job, not for the reported count. Different expectations over the day are covered with several heartbeats.

The grace period on the heartbeat, zero to 3,600 seconds, only applies to a missing ping and absorbs the counting job finishing a minute late. The alert for too few orders comes with the ping itself, without any waiting time.

Day, night and weekend: a schedule instead of a rigid rhythm

A fixed hourly rhythm also expects a report at three in the morning, and a minimum count that fits during the day triggers false alarms at night. XICTRAQ® heartbeats therefore support two expectation models: a fixed interval between one minute and 24 hours, or a cron schedule with its own time zone. With a cron schedule, the next due time is calculated in the heartbeat's time zone; if the ping has not arrived by then plus the grace period, the alert follows.

The expression follows the crontab format with five time fields for minute, hour, day, month and weekday, where 0 stands for Sunday in the weekday field (POSIX). A hyphen describes an inclusive range. 5 8-21 1-5 therefore means: Monday to Friday at 8:05, 9:05 and so on until 21:05. After the 21:05 run, the heartbeat expects the next ping only at 8:05 on the following weekday, so nights and weekends stay quiet.

Because the minimum count is fixed per heartbeat, each phase with its own expectation gets its own heartbeat and its own line in the server's crontab. A pattern for a mid-sized shop, with sample values:

# Crontab of the shop server, server time zone: Europe/Berlin
# minute hour day month weekday  command
5 8-21 * * 1-5   /opt/shop/count-orders.sh orders-weekday-x7k2m9qa 60
5 9-22 * * 0,6   /opt/shop/count-orders.sh orders-weekend-p4n8c2tw 60
5 7 * * *        /opt/shop/count-orders.sh orders-night-h3v6r1ze 480

Partial outages: when only one payment method fails

A zero alert detects a complete outage. Often, however, only part of the checkout fails, and the number drops without reaching zero. According to the Bundesbank's payment diary, internet payment services account for 50 % of payment transactions on the internet (Bundesbank), followed by direct debits and credit transfers at 30 % and credit cards at 15 % (Bundesbank). If one method fails, a share of roughly this size goes missing while the rest continues. Whether customers simply switch payment method is far from certain: for 54 percent of respondents, payment options are very important when choosing an online shop (Bitkom).

There are two ways to handle partial outages. A higher minimum count only works if the window contains enough orders for the loss of a third to stand out from chance, and it increases the number of false alarms. More precise is a separate heartbeat per payment method: the counting job filters by payment method and reports each one to its own heartbeat, with the rule of thumb applied per payment method. Which payment methods have which share in your own shop is shown in the article on analysing payment methods.

• Per shop or domain, if one system serves several countries or brands. If only the Austrian checkout fails, the total stays inconspicuous.
• Per sales channel, such as your own shop, a B2B portal and marketplace imports. A stalled import looks like a quiet day in the total.
• Per order route, if orders also arrive through an app or an interface.
• Not per product or category. There λ is usually too small, and the alerts turn into noise.

Order heartbeat, analytics alert and browser test compared

The order heartbeat is not the only way to notice a checkout failure. Three methods complement each other because they measure at different points. Browser monitoring with a checkout flow regularly clicks through the purchase process and sees a defect without real purchase traffic, but usually only up to the real payment; how such flows are built is shown in the article on the login flow and checkout. The analytics alerts of the cookieless analytics evaluate hourly what the tracker counted on the order confirmation, with a fixed threshold or as a drop compared with the same time window a week earlier. The order heartbeat counts in the database itself.

It gets revealing when two methods disagree. If the analytics alert reports a drop while the heartbeat sees normal numbers, the confirmation page or the tracking has a problem. If both drop, sales themselves are affected. If only the heartbeat goes missing, the fault lies with the counting job or its server. Automatic evaluations additionally point out unusual deviations, as described in the article on anomaly detection.

When the alert arrives: the first steps

An order alert says that something is wrong, not what. First confirm and narrow down the finding, ideally in a fixed sequence that is ready to hand for whoever is on call:

1. Place a test order with a payment method that can be cancelled. If it fails, the finding is confirmed, and the error message in the browser is often the most important clue.
2. Check payment methods one by one. If invoice works but card payment does not, the problem lies with the payment module or the payment provider.
3. Review changes from the last few hours: deploy, plugin update, new price or shipping rule. How a rollback with verification works is described in the article on deploy errors as a cause of outages.
4. Check the error log and the payment provider's callbacks. Rejected callbacks explain orders that were created but not confirmed.
5. Let it count after the fix. The next run with a sufficient count resolves the alert automatically; if you do not want to wait, start the counting job manually.

Who receives the alert at 8 p.m. on a promotion day and who takes over if nobody responds should be settled in advance. How to set thresholds and channels so that alerts are taken seriously is covered in the article on alarm fatigue. If a disruption lasts longer, a short note on the status page helps before the first customer emails arrive.

Set it up before Black Week, not during Black Week

For Black Friday and Cyber Monday 2025 the German retail federation expected sales of 5.8 billion euros (HDE), and according to a survey commissioned by the federation, 54 percent of the online shoppers surveyed also use Black Friday for Christmas shopping (HDE). Activating a new monitor only in that week means getting to know its false alarms in the week with the least time for them. A lead time is better:

1. Four weeks before: analyse orders per hour and weekday, determine λ per phase, set windows and minimum counts.
2. Three weeks before: set up the counting job, compare the count with the order list, create heartbeats with cron schedule and time zone.
3. Two weeks before: cross-check with a manual items=0. Does the alert reach the right person in the right channel? Then send a healthy ping; the alert must resolve.
4. Before the promotion days: with a clearly higher order rate, create an additional heartbeat with a higher minimum count that the cron schedule only expects on those days, for example 5 8-23 27-30 11 * for 27 to 30 November.
5. After Black Week: every false alarm points to a window that is too short or a minimum count that is too high.

How the website itself gets through these weeks is described in the article on Q4 traffic peaks linked at the beginning. If you keep your promotion page at the same address every year, the uptime check also gets a permanent address; see the Black Friday page with a permanent URL. The same technique also monitors backup runs, as the article on backup monitoring with heartbeats shows.