Incidents
Incidents bundle related alerts into a single event. They are automatically created for critical alerts (downtime of the website, a subpage or a service, timeout, SSL errors, safe browsing, blacklist, security score drop, defacement, performance budget) and can be manually created, acknowledged, resolved and documented. The timeline shows the full history — from first alert through automatic diagnostics to postmortem.
Automatic creation
An incident is created automatically as soon as an alert of one of the following types fires: downtime (downtime/down), unreachable monitored subpage (subpage_down), unreachable service on a port of the website (service_down), timeout, invalid SSL certificate (ssl_invalid), safe browsing warning, blacklist hit, security score drop, possible defacement or performance budget violation. Other alert types (e.g. SSL expiry warning, SEO change, slow response) only produce an alert without an incident — view and manage those under /app/alerts (view, acknowledge, resolve, delete). If an unresolved incident of the same kind for the same website already exists and started within the last 24 hours, the new alert is linked to it (timeline event "Alert added") instead of opening a second incident. This keeps a longer-running event as a single record. There are three kinds: availability (downtime, timeout, invalid SSL, subpage, service), security (safe browsing, blacklist, security score drop, defacement) and performance budget. If the security score drops during an outage, that gets an incident of its own, which does not keep the outage incident open. On automatic creation the severity is derived from the alert type: downtime, timeout, invalid SSL, safe browsing and defacement yield "Critical"; subpage, service, blacklist, security score drop and performance budget yield "Warning". The title combines the alert type with the affected website URL.
Auto diagnostics for availability incidents
When an incident is triggered by an outage, a timeout or a “down” state, XICTRAQ immediately captures a lightweight snapshot and attaches it as a timeline event “Auto diagnostics”: DNS resolution (A/AAAA, plus whether it resolved at all — if the name server does not answer at all, it reads “not measured” rather than “not resolved”), an HTTP probe (status code, total and connect time in milliseconds, primary IP and selected response headers such as server, content-type, cf-ray, location), a traceroute (tracepath, up to 12 hops, showing where the path breaks) and — where available — a screenshot of the error page. Diagnostics run only after the alert has been sent and never delay the notification. They give you immediate triage context: DNS problem, 5xx or a broken network path?
Create manually
Via the "Create" button on the incidents page you create an incident manually at any time — with a title (up to 300 characters) and severity (Critical, Warning or Info), optionally assigned to a website. Useful for planned maintenance or external events not triggered by an automatic alert. A manually created incident starts in state "Open" and gets a timeline event "Opened" flagged as "manual".
Lifecycle
An incident passes through the following states: **Open** — Just detected, not yet acknowledged. Escalation running. **Investigating**, **Identified** and **Monitoring** — optional intermediate states as used by professional status tools. You set them via a status update; each change is recorded as a "Status update" event with an optional public note (up to 2000 characters) in the timeline. **Acknowledged** — Someone has seen and taken ownership. Sets acknowledged_at and acknowledged_by on the incident. Stopping any running escalation is tied to the alert itself (acknowledging the alert or its ACK link via alerts.is_resolved), not to the incident status. **Resolved** — Problem fixed. Optionally with root cause (up to 500 characters) and postmortem (up to 20,000 characters). All linked, still-open alerts are closed alongside through the central lifecycle service, so the audit trail (delivery_log) stays consistent.
Timeline & comments
Every action is logged as an event in the timeline: opening, auto diagnostics, added alerts, status updates, comments, acknowledgement, edits, manual and automatic resolution. Each event carries a type, an optional actor and a timestamp. Comments (up to 2000 characters) can be added at any time — for internal communication or later documentation.
Edit & delete
Title and severity can be changed at any time (e.g. upgrading from Warning to Critical). An incident can be fully deleted — this clears the incident_id on all linked alerts, removes the complete timeline (incident_events) and deletes the incident record itself.
Status update templates
Three templates are built in (investigating, identified, monitoring). On top you save your own templates — up to 30 per account, each with a name, optional status (investigating, identified, monitoring, resolved) and predefined text (up to 2000 characters). Routine updates are inserted with one click instead of being retyped each time.
Auto-resolve on recovery
As soon as no linked alert of an incident is open any more (website reachable again, SSL renewed, score recovered), the incident is automatically set to “Resolved” — with the root cause “Auto-resolved” and a timeline event “Auto-resolved”. This applies to every resolve path (UI, API, ack token, AI assistant, automatic recovery), not just bulk resolution.
Cleanup cron
A cleanup job closes two things: first, incidents whose alerts are all resolved by now (idempotently via the same lifecycle service); second, cold incidents with no timeline activity for over 30 days. These are closed with the root cause “Auto-resolved (inactive for >30 days)”. In both cases an incident with a still-open alert stays open — the condition persists, e.g. a security score that is still down — and so does a manually created incident; both are only closed by hand.
Postmortem export
For resolved incidents a postmortem can be generated from a linked alert — structured documentation of the event for audits, customer reports or internal lessons learned. The download button (PDF) appears in the incident detail view once the incident is resolved; as Markdown the API returns it at GET /api/alerts/{id}/postmortem.md. A manually created incident has no linked alert and therefore no postmortem.
# API endpoints
All incident actions are available via the authenticated API. The list returns at most 200 incidents per request, newest first; page on with limit and offset, total gives the overall count. It can be filtered by status, severity and website (website_id).
GET /api/incidents?status=open Liste (max. 200 je Abruf, limit/offset)
POST /api/incidents Anlegen (title, severity, website_id?)
GET /api/incidents/{id} Detail + Timeline + verknüpfte Alerts
PATCH /api/incidents/{id} Titel/Severity/Status + optionale Notiz
DELETE /api/incidents/{id} Löschen (löst incident_id der Alerts)
POST /api/incidents/{id}/ack Bestätigen
POST /api/incidents/{id}/resolve Beheben (root_cause?, postmortem?)
POST /api/incidents/{id}/comment Kommentar (comment)
GET /api/incident-templates Eigene Vorlagen lesen
PUT /api/incident-templates Vorlagen ersetzen (max. 30)