What an A/B test actually measures during Black Week
An A/B test splits visitors at random between two or more variants and counts how often a conversion happens in each group. Two proportions are compared. If the samples are large enough, the binomial distribution can be approximated by the normal distribution, which yields a test similar to comparing two means (NIST/SEMATECH e-Handbook). XICTRAQ® uses this basis to compute the two-proportion z-test and reports the uplift with a 95% confidence interval (XICTRAQ feature page). How assignment, the z-test and sample size work in detail is covered in the introductory article on A/B testing without consent banners. This article deals with a different question: what a result from an exceptional week means.
Random assignment is the strength of the method. According to the NIST/SEMATECH statistics handbook, it is what makes conclusions from an experiment correct, unambiguous and defensible (NIST/SEMATECH e-Handbook). Because both variants run at the same time, any seasonal effect hits them equally: more visitors, higher willingness to buy, different times of day. Seasonal adjustment, as official statistics need it for time series, is therefore not required within a test. Eurostat describes seasonal adjustment as a method that removes recurring seasonal influences from a time series so that the development without the season becomes clearer (Eurostat). The German Federal Statistical Office uses such methods to remove effects such as the number of shopping days and public holidays from its index figures (Destatis).
What simultaneous assignment does not balance out is an interaction between variant and season. The test reliably answers whether A and B differ during the test period. Whether the same difference appears in a week without discounts is a transfer the test itself does not cover. Eurostat notes that seasonal effects differ not only between countries but also between individual indicators (Eurostat). The conversion rate of a promotion banner is one such indicator.
In experimental design this is called an interaction: the effect of one factor on the response depends on the level of another factor (NIST/SEMATECH e-Handbook). For a shop, this means the effect of variant B can depend on whether it is promotion week. The test then cleanly measures the effect during promotion week, and only that.
Why promotion week brings different buyers to the shop
Visitors during a promotion week are not the same people as in October, and they arrive with a different goal. Roughly one in two (52 percent) of the internet users surveyed wanted to use the reduced prices around Black Friday 2025, most of them online (Bitkom 2025). The survey covered 1,120 internet users in Germany aged 16 and over (Bitkom 2025). The figures on shopper types refer to those who intended to shop on Black Friday or during Cyber Week.
Most of those who want to shop on Black Friday or during Cyber Week say they plan their purchases.
— Bitkom, press release "Black Friday: Die Hälfte geht auf Schnäppchenjagd", November 24, 2025 (translated)
The breakdown shows what the mix looks like. 45 percent see themselves as "wish fulfillers" who treat themselves to long-desired items only at a sale price (Bitkom 2025). Just as many, 45 percent, are "rational buyers" who purchase everyday goods or larger items in a targeted way when they are on offer (Bitkom 2025). Only 28 percent count themselves as "impulse buyers" (Bitkom 2025), 22 percent as classic "bargain hunters" (Bitkom 2025). 33 percent use the discounts mainly for gifts (Bitkom 2025), and 61 percent already have a clear idea of how much they want to spend (Bitkom 2025). Multiple answers were possible.
For an A/B test, this mix is decisive. Someone who has been waiting for weeks for the price of a specific device to drop buys during promotion week largely regardless of whether the banner is green or blue. An impulse buyer, by contrast, reacts much more strongly to design, placement and urgency. If the share of these groups shifts, the effect of a variant shifts too, without the test being wrong. It correctly measures the effect on exactly this audience.
On top of that, the season itself varies from year to year. In October and November 2024, online sales of goods slumped by 4.2 percent in the middle of the Christmas season, Black Friday promotions included (bevh). In the same period of 2025, consumers spent 2.9 percent more online (bevh). And within the same season, product categories drift apart: furniture and home furnishings grew by 9.1 percent in October and November 2025 (bevh), while apparel and shoes grew by only 0.1 percent (bevh). A result from the 2026 promotion week is therefore initially a hypothesis for the 2027 promotion week, not a fixed value.
Four reasons a winner can lose in January
The buyer mix and the conditions of promotion week produce four typical mechanisms. Each of them can create a real effect during promotion week that turns out weaker, disappears or even reverses in normal weeks.
Planned purchases dominate promotion week. A variant that persuades undecided visitors has less leverage then; a variant that stresses savings hits the very reason for the visit. In January, different visitors arrive with different questions.
When the price drops noticeably, the price carries the decision. Design details work differently next to it than in weeks without discounts. A variant that highlights a strikethrough price simply has nothing to highlight once the promotion is over.
People waiting for the sale day look at products several times beforehand and buy later in a new session. This dilutes the measured difference, as the next section shows.
Under load, heavy variants with large images or extra scripts load more slowly. What gets measured is then design plus load time under load. How shops prepare for this is covered in traffic peaks in Q4.
Discount pressure also has a legal side. Anyone announcing a price reduction on goods to consumers must state the lowest total price they applied in the 30 days before the reduction (PAngV, Section 11(1)). If you test different price displays during promotion week, both variants have to meet this obligation; the details are described in the article on strikethrough prices on Black Friday. The German e-commerce association also describes how strongly discounts shape buying behavior: many gifts were only bought at a "subsidized price", according to the deputy managing director of bevh on the 2024 Christmas season (bevh).
A countdown follows a similar pattern. A timer that shows the end of a real offer can have a measurable effect during promotion week because the offer actually ends. In a week without a deadline there is nothing to count down to, and the test result cannot be transferred at all. If you test countdowns, also keep the rules for time limits in the cart in mind.
Repeat visits dilute the difference
How much returning visitors affect the result depends on how a tool assigns visitors to variants. A/B assignment in XICTRAQ® works without cookies and without storing anything on the device. Assignment is tied to a session key built from the IP address, a random value generated anew each day and the date. Within a single day, the same visitor therefore sees the same variant (XICTRAQ documentation). On the next day a new session is created and assigned again at random. The same applies when the connection changes, for example from mobile data to home Wi-Fi, and to visitors who stay in the shop past the change of day.
In an ordinary week this often matters less, if fewer purchases are prepared over several days. During promotion week the pattern is an obvious one: compare on Monday, buy on Friday. Many online shoppers also use several devices in general: 73 percent of them shop on a smartphone and 52 percent on a notebook (Bitkom 2025), together well over 100 percent. For the test, this creates four effects:
- The purchase counts for the variant of the buying session. If variant B persuaded on Monday and the Friday session was assigned variant A, the conversion counts for A.
- The persuading session counts as a visit without a purchase. The Monday session with variant B appears in the report without a conversion.
- Purchases outside the test page drop out. If the buying session does not load the page with the experiment at all, for example because the visitor goes straight from a wish list to checkout, the purchase counts for no variant.
- The same person enters several times. Each new daily session is a new entry in the population, even if the same person is behind it.
A simple calculation shows the order of magnitude. Suppose variant B really raises the purchase probability by 10 percent. If the decision for 40 percent of purchases is made in an earlier session, those purchases spread randomly across both variants and add nothing to the difference. What remains measurable is roughly 6 percent uplift (0.6 × 10). Because the required sample is roughly inversely proportional to the square of the difference you are looking for, the test then needs about 2.8 times as many sessions (1 / 0.6²). All values in this calculation are assumptions. The direction is typical: repeat visits pull the measured effect toward zero. The sample size calculator shows what a smaller minimum effect means for the number of sessions you need.
A/B assignment in XICTRAQ® deliberately does not recognize visitors across days. That is what makes testing without storing anything on the device possible, and it is also the reason for the dilution. Because of it, the measured difference tends to be smaller than the effect on the individual purchase decision. A small or missing difference during this week is thus no proof that a variant has no effect.
Pause, extend or read as a seasonal segment
For every test that runs into the promotion period, a deliberate decision before the promotion starts pays off. Which option fits depends on what the result will be used for later. The following overview sorts the most common situations.
| Situation | Recommendation | In XICTRAQ® |
|---|---|---|
| Test has been running for weeks, the result is meant for the whole year | Pause it for the promotion period or leave the promotion days out of the analysis | Status "paused", then resume; set the analysis period to the weeks before and after |
| Test only concerns promotion content such as a banner or landing page | A separate experiment with its own name that runs only during the promotion period | Create a new experiment and end it after the promotion |
| Test keeps running, the result should show both situations | Read the promotion period as a separate segment instead of mixing it with normal weeks | Choose the analysis period per window and put the intervals side by side |
| Planned sample size not yet reached before the promotion | Do not top it up with promotion days; resume or restart after the season | Recalculate the sample with a smaller minimum effect |
| Variant changes price display or discount presentation | Only test if both variants meet price indication rules | Use the result only for weeks with discounts |
Pausing and ending differ considerably in XICTRAQ®. Paused and ended tests no longer assign new visitors (XICTRAQ documentation). With a paused test, the script on the page receives no variant, so the page shows what it shows without a variant, usually the original. Conversions from sessions assigned before the pause still count. An ended test, by contrast, is closed: it cannot be resumed, and conversions after the end no longer count. If you want to keep testing after the promotion, pause the test instead of ending it.
Extending in this situation does not mean letting the test run until Christmas. December is also a season, with gift purchases, delivery deadlines and holidays. Weeks that match a shop’s normal operations often only come around again in the new year, depending on the product range. If you need a test for the promotion, set it up as a separate experiment and name it so the period is part of the name:
// Separate experiment for the promotion period only
xictraq.experiment('promo-banner-bw-2026').then(variant => {
// without a running test no variant is returned: the original stays
if (variant === 'B') {
document.querySelector('.promo-banner')?.classList.add('promo-banner--b');
}
});
// on the order confirmation page
xictraq.convert('promo-banner-bw-2026');The name records in the report what the result stands for. Nobody will mistake a test called "promo-banner-bw-2026" for a year-round result. Once the promotion is over, it is ended, and a new test with its own sample size planning starts for normal weeks if needed.
Analyze normal weeks and the promotion period separately
If a test runs straight through the promotion period, it can still be read cleanly. In the A/B report of XICTRAQ®, the table follows the selected date range: it counts the assignments within the range and their conversions up to its end. Each variant shows visitors, conversions, conversion rate, uplift with interval and p-value. This turns the promotion period into a separate segment that can be placed next to the weeks before and after. Here is how to proceed:
- Note the shop’s promotion period with dates, for example November 20 to 30, including early offers for newsletter subscribers.
- As the first window, select the normal weeks before the promotion and note the uplift interval.
- As the second window, select exactly the promotion period and write the interval next to it.
- After the promotion, analyze a third window once enough sessions have accumulated.
- Look at the whole interval for each window, not just the winner label.
- When reading several windows, apply a stricter threshold to the p-value.
The chart above shows what such a picture can look like. In the normal weeks, the uplift of B is +13 percent with an interval from +3 to +23 percent; in the promotion period it is −2 percent with an interval from −11 to +7 percent. These are example values. Read them like this: in the normal weeks the entire interval lies above zero, so an advantage of B is proven there. In the promotion period the interval includes zero, so no advantage is proven there. The fact that the two intervals overlap means neither that the effect is the same in both periods nor that it differs. For the decision, the separate reading is usually enough: adopt B for normal weeks and treat promotion week as a case of its own.
Reading several windows means making several comparisons, and with each one the chance rises that one of them stands out by accident. The Bonferroni method allows multiple comparisons and intervals while maintaining an overall confidence level (NIST/SEMATECH e-Handbook). To do so, each individual comparison is set more strictly: with three windows and an overall error rate of 5 percent, the threshold per window is about 0.017 (0.05 / 3). In XICTRAQ®, a variant is marked green as the winner once the p-value drops below 0.05 and the uplift is positive (XICTRAQ documentation). If you read three windows, check the displayed p-value against the stricter threshold yourself.
Experimental design has a term for this: the block. It serves to isolate a systematic effect so that it does not obscure the main effects (NIST/SEMATECH e-Handbook). The promotion period is such a block. A result from it applies to weeks with a promotion, a result from the normal weeks to weeks without one. Reporting both separately loses no information and avoids the wrong transfer.
What belongs in the report after Black Week
A test report after promotion week is useful if it is still understood next year. That takes more than the winning variant. Costs and revenue per campaign belong in the ROAS report for Black Week, and the comparison with the previous year needs the calendar shift of Black Friday. For the experiment itself, a few details are enough:
- Name of the experiment, variants and exact runtime with dates
- The shop’s promotion period and which discounts ran during that time
- Per period: sessions, conversions, uplift with interval and p-value
- A note that assignment is per daily session and returning visitors count several times
- Technical anomalies during the test period, such as longer load times under load
- Decision with scope: adopted for promotions, adopted for normal weeks, or to be retested
Before a result goes into the report as a design effect, it is worth checking technical causes. A button that does not respond under load shows up in rage and dead clicks, a checkout where one payment method fails shows up in the payment method analysis. Whether the site was reachable throughout promotion week at all is shown by uptime monitoring. A variant that loses only because its image does not load under load is not a design finding.
Decide on your tests before the promotion starts
Only a few days lie between the publication of this article and Black Friday. Go through every running test once and decide: pause it, run it as a separate promotion experiment, or let it continue and read it as a segment. Note the shop’s promotion period with dates before the first discount starts, so the windows fit exactly later. And for new tests, plan for a smaller measurable effect if many of your customers plan over several days.
A/B testing is part of cookieless analytics and available as a premium module from the Business plan (XICTRAQ documentation); the plans are listed on the pricing page. The documentation on A/B tests and insights describes assignment, status and analysis in detail, and the live demo lets you look at the report without signing up. Reading the promotion period as a separate segment keeps both: the finding for Black Week and a clean baseline for the weeks after, measured with analytics that needs no banner.
This article is based on: NIST/SEMATECH e-Handbook of Statistical Methods, sections 7.3.3 (comparing two proportions), 5.7 (glossary of experimental design: interaction, randomization, blocking) and 7.4.7.3 (Bonferroni method); Eurostat, glossary "Seasonal adjustment"; Destatis, glossary "Kalender- und saisonbereinigte Werte"; Bitkom, press release "Black Friday: Die Hälfte geht auf Schnäppchenjagd" of November 24, 2025, representative survey by Bitkom Research among 1,120 internet users aged 16 and over in Germany, calendar weeks 40 to 41 of 2025; bevh, press releases of December 10, 2024 and December 11, 2025 based on the consumer survey "Interaktiver Handel in Deutschland" with 40,000 respondents aged 14 and over per year; German Price Indication Ordinance (PAngV), Section 11; XICTRAQ documentation on A/B tests and insights; XICTRAQ feature page Cookieless Analytics. The curves, intervals and the dilution calculation are based on assumed values.