Testing your alerting chain: does the alert arrive at night?

Why a green test button says little about the night

A typical sequence: during setup in spring, someone clicks "Send test alert" in the notification settings, every channel reports success, and the topic is closed. Over the summer, the Slack channel for incidents is archived and replaced with a new one. Since a rebuild, the webhook for the ticket system points to an address that no longer exists. And the colleague on call is away over the long weekend without the on-call schedule knowing about it. This only comes to light on the night the checkout fails and the notification goes nowhere.

The test button is still useful; it just checks something else. In XICTRAQ® it sends a test alert through every active channel and returns a result per channel with status, HTTP code, error and number of attempts. This test alert is not stored. It therefore has no delivery log attached to an alert, triggers no escalation, reaches no on-call person and cannot be acknowledged. And it deliberately bypasses two settings: the choice of which alert categories arrive on which channel, and the quiet hours per channel. A test is meant to show whether a channel delivers.

That leads to an uncomfortable insight: a green test at 2:14 a.m. does not prove that a real warning alert at 2:14 a.m. arrives. If the channel is in quiet hours, XICTRAQ® delivers only critical alerts there right away; the rest arrive as a digest once quiet hours end. That is intended but has to be a deliberate decision.

What BSI Standard 200-4 means by an alerting drill

BSI Standard 200-4 on business continuity management treats the alerting drill as a separate type of exercise. It is meant to identify sources of error and weaknesses in alerting, determine how effective the alerting procedures are and verify the response time more concretely (BSI Standard 200-4). Response times are to be determined and documented along the way. The key sentence of the section:

Alerting drills should be used to test whether the organizational and technical measures ensure alerting within the required time frame (also outside normal business hours).

— BSI Standard 200-4, section 13.8 Alerting drill (translated from German)

The standard distinguishes two forms. Technology-oriented tests check whether the means and procedures of communication work. Application-oriented exercises check the alerting documentation and the organizational rules, such as whether role holders and their deputies can be reached; in addition, the speed of the response can be tested and practiced (BSI Standard 200-4). Applied to website monitoring, clicking the test button is a technology-oriented test. The application-oriented drill needs an alert that takes the same path as a real one.

For an online store, the standard is not mandatory. For organizations that build business continuity management at the "reactive BCMS" level, however, alerting drills are to be held regularly (BSI Standard 200-4). The standard is useful anyway because it proposes a measurable target. As an example target, the standard names a maximum of 30 minutes from the initial alert until all required role holders have positively acknowledged it (BSI Standard 200-4). You can adopt that value or adjust it to your own on-call setup. What matters is that it is fixed before the drill.

Where alerting chains break at night

Breaks typically do not happen during setup but weeks later, through changes that have nothing to do with monitoring. A channel is cleaned up, a token renewed, an endpoint moved, a phone set up again. Nobody checks at that moment whether an alert still arrives afterward. Six typical breaking points:

With Slack, Telegram and your own webhooks, a break leaves an error code in the delivery log, which only someone who looks after the alert will see. With a phone in do-not-disturb or a colleague on vacation there is no error code at all, because the message was delivered. These cases usually surface only in a drill that runs to acknowledgment. The retry logic section in configuring monitoring alerts explains that channels themselves can fail and why XICTRAQ® therefore repeats delivery attempts.

Email, chat and push: what the channels limit themselves

In XICTRAQ®, email is the default channel for all categories and at the same time a slow one as soon as something goes wrong. If a mail server cannot deliver a message right away, it has to wait before trying again. The SMTP standard generally calls for an interval of at least 30 minutes (RFC 5321). Retries continue until the message is delivered or the sender gives up, and the give-up time should generally be at least four to five days (RFC 5321). A single temporary error on the recipient side can thus use up the entire time budget of an alert. To the sender, the handover to the mail server still looks like success.

Then there is attention. The BSI standard considers asynchronous means of communication less suitable for alerting because the sender cannot know whether and when the recipient receives the information. In its assessment, alerts by email often get lost among other emails; calls to mobile phones or alerting apps are better suited (BSI Standard 200-4). XICTRAQ® does not place calls or send text messages; it delivers via chat services, email, webhook and browser push. That makes acknowledgment and escalation all the more important: they turn an asynchronous message into a chain with feedback. Email belongs in that chain as a record and a fallback; at night, a different channel should do the waking.

Chat services deliver fast but set limits. If a bot is throttled, the API answers with status 429, which according to the HTTP standard indicates that too many requests were sent in a given amount of time (RFC 6585). In that case XICTRAQ® repeats the delivery after a short interval and writes the result per channel to the delivery log.

Browser push depends on the device. XICTRAQ® enables it per device, not per monitor; a laptop that sits closed in a backpack at night is not a recipient. Under the Web Push standard, the push service must not deliver a message once the time-to-live specified by the sender has elapsed (RFC 8030). Whether a push really appears on the on-call phone at night is therefore shown only by a drill on exactly that device.

Quiet hours and severity: what should get through at night

Before you drill, it has to be clear which alert should wake anyone at all at night. The article on building an on-call rotation without burnout separates alerts that need someone right away from those that can wait until morning. In XICTRAQ®, four factors decide what arrives where:

• Categories per channel: alerts are grouped into categories such as uptime, SSL, security and system; for each category you choose the channels.
• Quiet hours per channel: during quiet hours, a channel holds back non-critical alerts and then sends them as a digest, unless you have turned the digest off. Critical alerts get through.
• Severity of the alert: XICTRAQ® assigns the severity when the alert is triggered; whether an alert counts as critical decides whether it breaks through quiet hours.
• Escalation policy: a policy can be restricted to specific websites and a minimum severity. Alerts without a website, such as those from heartbeats, do not match a policy with a website selection; for them, a policy that filters only by severity or the default policy applies.

This leads to two questions that are often confused. The first is: does a critical alert reach the on-call person at night, and does that person acknowledge it within the target time? That is the question for the drill. The second is: does a warning stay as quiet at night as intended? That is a configuration question you settle in the settings before the drill. To make sure only confirmed outages wake people at night and not every brief glitch, it is also worth looking at flapping alerts and how to tame them.

Setting up the drill: a real alert from a drill monitor

The application-oriented drill needs an alert that is real but does not report real damage. The simplest way is a dedicated heartbeat monitor used only for drills. Besides the normal ping, heartbeats have an address for actively reporting a failure: a call to /fail sets the monitor to "failed" and immediately triggers an alert of type heartbeat_failed without waiting for the grace window. This alert has critical severity, is stored, passes through categories, quiet hours and escalation like any other alert, and can be acknowledged.

So that the drill monitor does not raise an alarm on its own between drills, it needs a regular healthy ping, for example from an existing cron job once a day around noon. Create it with a matching schedule and give it a name that is instantly recognizable as a drill in the alert, such as "Drill alerting chain". If you already use heartbeats for exports, for example for the product feed, simply add the drill monitor next to them. At drill time, a single call is enough:

# Trigger the drill alert (replace YOUR-SLUG with the slug of the drill monitor)
curl -fsS -G --data-urlencode "msg=Drill alert, please acknowledge" \
  "https://api.xictraq.com/api/heartbeat/YOUR-SLUG/fail"

# After acknowledgment: healthy ping, the alert is resolved
curl -fsS "https://api.xictraq.com/api/heartbeat/YOUR-SLUG/ping"

After acknowledgment, close the alert with a healthy ping. The monitor recovers automatically, the alert is resolved, and an all-clear goes to the channels that received the alert. That is part of the drill too: if the all-clear does not arrive, the on-call person will not learn in a real incident that it is over.

Running the alerting drill in seven steps

The drill itself takes only minutes; the only real effort is the preparation. This order is practical:

1. Set the target: for example acknowledgment within 30 minutes of the alert, based on the example target (BSI Standard 200-4), and which channels must deliver within that time.
2. Choose the time: deliberately inconvenient, that is at night, on a weekend or on a public holiday. The standard also mentions the lunch break and the time shortly after work and advises informing the relevant parties beforehand for such dates, for example the staff or works council (BSI Standard 200-4).
3. Check the channels technically beforehand: press "Send test alert" the day before. Whatever fails here does not have to be found at night.
4. Trigger and note the time: trigger the drill monitor via /fail. The time appears in the alert timeline.
5. Watch delivery: the alert detail view lists the delivery channels with their result. If an escalation stage kicks in, record who was notified and when.
6. Have it acknowledged: the on-call person confirms the alert via the link in Slack, Teams, Discord or the email, without having to log in, or in the alert overview. That stops the escalation.
7. Resolve and evaluate: healthy ping, wait for the all-clear, compare the result with the target and record every deviation as an action item with an owner.

The standard reports that unannounced alerting drills have proven themselves in practice because they come closest to reality. To keep participants cooperative, it can be announced in advance that a drill will take place within a certain period (BSI Standard 200-4). German working time law is relevant here as well: under Section 5 of the Working Hours Act, employees must have an uninterrupted rest period of at least eleven hours after the end of their daily working time (ArbZG). How a nighttime drill alert during on-call duty affects that is something to clarify with HR before the first night drill.

Evaluating: delivery log, time to acknowledge, gaps

For the evaluation, the BSI standard recommends using the logs of the alerting software and combining them with the notes taken during the drill (BSI Standard 200-4). Alerting drills allow quantitative evaluation, for instance as "passed" or "failed" or as a comparison with a time target (BSI Standard 200-4). In XICTRAQ® the record sits on the alert itself: the delivery log lists status, HTTP code, error and number of attempts per channel, and the timeline shows when the alert was triggered, acknowledged and resolved. From the alert overview you can download a postmortem that serves as the basis for the drill report.

• time of the alert and of the acknowledgment, and the resulting time to acknowledge
• per channel: delivered, error with code, or not set up
• escalation stages triggered and who was notified at each
• device and channel through which the on-call person actually noticed the alert
• arrival of the all-clear per channel

Every alerting drill has a limit, and the standard states it openly: it checks only part of the response time, because the time until an event is detected is not included and can ultimately only be estimated or derived from real events (BSI Standard 200-4). In monitoring, that detection time depends on the check interval and the grace period before the outage alert. How long an attack can go unnoticed depending on the signal is covered in attacks noticed too late: detection time per signal.

Typical findings and what helps

The following overview sorts typical findings of a first drill by what becomes visible during the drill, the usual cause and the remedy.

For email findings, the free mail check shows whether SPF, DKIM and DMARC of a domain are correct. Do-not-disturb, vacation and an escalation stage without a recipient, on the other hand, usually come to light only in a drill.

Cadence: when to drill and who needs to take part

The BSI standard does not prescribe a fixed cadence for the alerting drill. It calls for annual planning in which all defined alerting paths are tested over a set period (BSI Standard 200-4). For website teams, a simple split works well:

• After every change to a channel: test button. The documentation names switching a Slack channel, a new bot token or a changed webhook URL as occasions.
• Before peak periods: a drill with a real alert, for example four weeks before Black Week and the holidays, so there is time for corrections.
• After changes in the on-call team: when someone joins, swaps a phone or the schedule is rebuilt.
• Occasionally outside business hours: at night or on a weekend, as the standard recommends for on-call teams reachable outside working hours.

Everyone who is part of the chain in a real incident needs to take part, including the deputies. Vacation periods and public holidays are typical points where schedules are wrong. XICTRAQ® resolves the person on duty live from the on-call schedule; you enter exceptions and vacation cover as an override with a time period and a substitute. A drill during the vacation season shows whether that is actually maintained. How escalation stages are built in principle is described in incident management and escalation.

A rehearsed alerting chain before Black Week and the holidays

An alerting chain is only checked once it has run completely at an inconvenient time: alert, delivery, escalation, acknowledgment and all-clear. The test button remains the tool for a quick check after every change. Schedule the drill four weeks before Black Week and the holidays, and you have time to fix archived channels, expired tokens and gaps in the on-call schedule before they become costly in a real incident.

All channels, escalation stages and on-call schedules are described on the alerting and notifications page. Escalation policies and on-call schedules are available from the Business plan; the overview is on pricing and plans. You can try the drill with a free account, which already includes heartbeats.