A/B testing & automated insights
Statistically valid A/B tests start with three lines of JavaScript — significance based on a real two-proportion z-test, no hand-waving. In parallel, XICTRAQ scans your data hourly for statistically significant anomalies — traffic drop, traffic spike, bounce increase and more — and pushes “automated insights” as dismissible cards to your dashboard.
# A/B testing
Assign variant, report conversion — done. No visual editor, but honest statistics with uplift and p-value.
Three lines of JavaScript
Assign variant, fire conversion on click — that's all. Both calls are part of the unified tracker (no extra snippet); no WYSIWYG editor, no code-injection risk. The assignment is fetched via “POST /api/p/ab”, the conversion is reported via “POST /api/p/abc”.
xictraq.experiment('cta_button_v2').then(variant => {
document.getElementById('cta').textContent =
variant === 'B' ? '14 Tage gratis testen' : 'Jetzt starten';
});
document.getElementById('cta').addEventListener('click', () => {
xictraq.convert('cta_button_v2');
});Two-proportion z-test with p-value
Instead of just “X wins”, XICTRAQ shows the full statistical result per variant: visitors, conversions, conversion rate, uplift (relative improvement over the first variant as baseline) and p-value. The p-value comes from a two-sided two-proportion z-test with pooled standard deviation (normal-distribution approximation per Abramowitz & Stegun). A winner is marked green in the app once the p-value drops below 0.05 (95 %) and the uplift is positive.
Sample-size calculator
Before the test, XICTRAQ computes the required sample size from your baseline conversion rate, the desired minimum detectable effect (MDE, relative uplift) and selectable confidence level (90/95/99 %) and statistical power (80/90/95 %, default 80 %). It outputs the required visitors per variant, the total (A + B), the expected target CR and an estimated test duration at 100 / 500 / 2,000 / 10,000 visitors per day. This prevents premature stopping when the test is not yet conclusive.
Up to five variants, sticky without a cookie
An experiment has two to five variants (e.g. Control, Variant A, Variant B). Assignment is sticky via a salt-hashed session key (sha256 of IP + salt + day) — within a single day the same visitor sees the same variant, with no cookie at all. Each experiment runs in status running / paused / ended; paused and ended tests no longer assign new visitors. Detected bot/datacenter traffic is neither assigned nor counted as a conversion.
# Automated insights
An hourly cron job searches for statistically significant anomalies in your analytics data. Findings appear as cards in the dashboard with a concrete description.
How the comparison works
The basis is a same-weekday comparison: yesterday's value is set against the most recent same weekdays of the past 28 days (e.g. Saturday against the last Saturdays), so weekend dips don't trigger a false alarm every Monday. Mean and standard deviation are derived from these samples and a z-score is computed. So small sites don't constantly beep, a minimum basis applies: at least 14 days of history, at least three same weekdays and an average of at least 50 page views on that weekday. Detected AI bots are excluded before the comparison.
What XICTRAQ currently detects
Eleven signal types, each with its own threshold and graduated severity (info / warning / critical): (1) traffic drop — yesterday at least −40 % below the typical same weekday and z ≤ −2.5 (critical from z ≤ −4). (2) traffic spike — at least +80 % and z ≥ 3. (3) bounce increase — bounce rate at least 15 percentage points higher and significant (z ≥ 2.5; critical from z ≥ 4). (4) error spike — at least 3 new, previously unseen JavaScript errors in the last hour (critical from 10). (5) referrer surge — one referrer domain brings a lot of traffic yesterday but was barely present the week before. (6) geo shift — yesterday's top country accounts for over 40 % and was not dominant the week before. (7) device-mix shift — the mobile share jumps by at least 25 percentage points. (8) slow response with falling traffic — average load time rose sharply while traffic dropped. (9) order drop — yesterday at least −40 % fewer recorded orders than on the typical same weekday and z ≤ −2.5 (critical from z ≤ −4); evaluated from an average of 8 orders on that weekday, and the spread is never taken to be smaller than the random noise of the order count. (10) revenue drop — the same thresholds for revenue, reported only if the order count itself did not drop. Both shop signals stay silent when traffic already dropped on the same day; (1) then reports the cause. (11) capture rate dropped (tracker_coverage_drop) — with the Shopware order import: the share of online orders captured by the tracker over the last three complete days is below 80 % of its value in the 28 days before (critical below 40 %), for example because a content blocker or a theme update affects the confirmation page; it is computed from the first tracker purchase, with at least 14 days of baseline, 15 online orders in the three days and 60 in the baseline, and the signal stays silent during a traffic drop.
Dismissible, debounced, delivered as an alert
Findings appear as dismissible cards in the dashboard. So the same finding doesn't reappear every hour, a debounce window is active per website and signal type (default 7 days, error spike 6 hours). Open insights older than 7 days are closed automatically — a persisting anomaly is re-reported afterwards anyway, a resolved one disappears. Findings of severity “warning” and “critical” are additionally delivered as an alert of type “Analytics anomaly” through your notification channels; you can unsubscribe from it under Settings › Notifications in the “Changes and analytics” category. When the card is dismissed or closed automatically after 7 days, the alert closes too, without an all-clear message.