Skip to content
GDPR compliant. Hosted in the EU.
14-day free trial →
A/B testingSeasonal segmentBlack Week
Analytics 18 min read 17 Nov 2026

A/B tests during Black Week: how to read the results

An A/B test from Black Week rarely shows what works in January: why purchase motive, discounts and repeat visits shape the result and how to read it correctly.

X

XICTRAQ® Team

XICTRAQ®

A/B-TestingStatisticsConversionBlack Week
28%
of Black Friday shoppers count themselves as impulse buyers (Bitkom 2025)
+9.1%
online sales of furniture and home furnishings in Oct. and Nov. 2025, apparel and shoes +0.1% (bevh)
+2.9%
online sales of goods in Oct. and Nov. 2025, after −4.2% a year earlier (bevh)
A/B test over three weeks: normal weeks vs. promotion periodDaily conversion rate for variants A and B (example values)Conversion rate per sessionVariant AVariant BShop’s promotion periodExample: Nov 20–301%2%3%4%5%0%Nov 10Nov 17Nov 24Nov 30Order flipsB ahead, then A aheadBoth variants run at the same time: the season hits A and B alike,but the gap between them depends on the period.Uplift of B vs. A95% interval per period+20%+10%−10%−20%overlap0%+13%−2%Normal weeksNov 10–19Promotion periodNov 20–30Overlap alone does not show whetherthe two periods differ.z-test, assignment per sessionMethod: NIST/SEMATECH e-Handbook, section 7.3.3 · all values are examples
Shortly before Black Week, quite a few online shops launch one more A/B test: a new promotion banner, a different order of payment methods, a more prominent button in the cart. Two weeks later the report shows a clear winner, and the variant is meant to stay for the rest of the year. That step is risky. An A/B test compares variants fairly as long as both run at the same time. Whether the result also holds in an ordinary week is a question it does not answer. This article shows why results from a promotion week rarely transfer, how repeat visits dilute the measured difference, and how to read the promotion period in XICTRAQ® as a separate seasonal segment.

What an A/B test actually measures during Black Week

An A/B test splits visitors at random between two or more variants and counts how often a conversion happens in each group. Two proportions are compared. If the samples are large enough, the binomial distribution can be approximated by the normal distribution, which yields a test similar to comparing two means (NIST/SEMATECH e-Handbook). XICTRAQ® uses this basis to compute the two-proportion z-test and reports the uplift with a 95% confidence interval (XICTRAQ feature page). How assignment, the z-test and sample size work in detail is covered in the introductory article on A/B testing without consent banners. This article deals with a different question: what a result from an exceptional week means.

Random assignment is the strength of the method. According to the NIST/SEMATECH statistics handbook, it is what makes conclusions from an experiment correct, unambiguous and defensible (NIST/SEMATECH e-Handbook). Because both variants run at the same time, any seasonal effect hits them equally: more visitors, higher willingness to buy, different times of day. Seasonal adjustment, as official statistics need it for time series, is therefore not required within a test. Eurostat describes seasonal adjustment as a method that removes recurring seasonal influences from a time series so that the development without the season becomes clearer (Eurostat). The German Federal Statistical Office uses such methods to remove effects such as the number of shopping days and public holidays from its index figures (Destatis).

What simultaneous assignment does not balance out is an interaction between variant and season. The test reliably answers whether A and B differ during the test period. Whether the same difference appears in a week without discounts is a transfer the test itself does not cover. Eurostat notes that seasonal effects differ not only between countries but also between individual indicators (Eurostat). The conversion rate of a promotion banner is one such indicator.

An interaction, not a nuisance

In experimental design this is called an interaction: the effect of one factor on the response depends on the level of another factor (NIST/SEMATECH e-Handbook). For a shop, this means the effect of variant B can depend on whether it is promotion week. The test then cleanly measures the effect during promotion week, and only that.

Why promotion week brings different buyers to the shop

Visitors during a promotion week are not the same people as in October, and they arrive with a different goal. Roughly one in two (52 percent) of the internet users surveyed wanted to use the reduced prices around Black Friday 2025, most of them online (Bitkom 2025). The survey covered 1,120 internet users in Germany aged 16 and over (Bitkom 2025). The figures on shopper types refer to those who intended to shop on Black Friday or during Cyber Week.

Most of those who want to shop on Black Friday or during Cyber Week say they plan their purchases.

— Bitkom, press release "Black Friday: Die Hälfte geht auf Schnäppchenjagd", November 24, 2025 (translated)

The breakdown shows what the mix looks like. 45 percent see themselves as "wish fulfillers" who treat themselves to long-desired items only at a sale price (Bitkom 2025). Just as many, 45 percent, are "rational buyers" who purchase everyday goods or larger items in a targeted way when they are on offer (Bitkom 2025). Only 28 percent count themselves as "impulse buyers" (Bitkom 2025), 22 percent as classic "bargain hunters" (Bitkom 2025). 33 percent use the discounts mainly for gifts (Bitkom 2025), and 61 percent already have a clear idea of how much they want to spend (Bitkom 2025). Multiple answers were possible.

For an A/B test, this mix is decisive. Someone who has been waiting for weeks for the price of a specific device to drop buys during promotion week largely regardless of whether the banner is green or blue. An impulse buyer, by contrast, reacts much more strongly to design, placement and urgency. If the share of these groups shifts, the effect of a variant shifts too, without the test being wrong. It correctly measures the effect on exactly this audience.

On top of that, the season itself varies from year to year. In October and November 2024, online sales of goods slumped by 4.2 percent in the middle of the Christmas season, Black Friday promotions included (bevh). In the same period of 2025, consumers spent 2.9 percent more online (bevh). And within the same season, product categories drift apart: furniture and home furnishings grew by 9.1 percent in October and November 2025 (bevh), while apparel and shoes grew by only 0.1 percent (bevh). A result from the 2026 promotion week is therefore initially a hypothesis for the 2027 promotion week, not a fixed value.

Four reasons a winner can lose in January

The buyer mix and the conditions of promotion week produce four typical mechanisms. Each of them can create a real effect during promotion week that turns out weaker, disappears or even reverses in normal weeks.

Different purchase motive

Planned purchases dominate promotion week. A variant that persuades undecided visitors has less leverage then; a variant that stresses savings hits the very reason for the visit. In January, different visitors arrive with different questions.

Discount pressure

When the price drops noticeably, the price carries the decision. Design details work differently next to it than in weeks without discounts. A variant that highlights a strikethrough price simply has nothing to highlight once the promotion is over.

Returning planners

People waiting for the sale day look at products several times beforehand and buy later in a new session. This dilutes the measured difference, as the next section shows.

Traffic peaks

Under load, heavy variants with large images or extra scripts load more slowly. What gets measured is then design plus load time under load. How shops prepare for this is covered in traffic peaks in Q4.

Discount pressure also has a legal side. Anyone announcing a price reduction on goods to consumers must state the lowest total price they applied in the 30 days before the reduction (PAngV, Section 11(1)). If you test different price displays during promotion week, both variants have to meet this obligation; the details are described in the article on strikethrough prices on Black Friday. The German e-commerce association also describes how strongly discounts shape buying behavior: many gifts were only bought at a "subsidized price", according to the deputy managing director of bevh on the 2024 Christmas season (bevh).

A countdown follows a similar pattern. A timer that shows the end of a real offer can have a measurable effect during promotion week because the offer actually ends. In a week without a deadline there is nothing to count down to, and the test result cannot be transferred at all. If you test countdowns, also keep the rules for time limits in the cart in mind.

Repeat visits dilute the difference

How much returning visitors affect the result depends on how a tool assigns visitors to variants. A/B assignment in XICTRAQ® works without cookies and without storing anything on the device. Assignment is tied to a session key built from the IP address, a random value generated anew each day and the date. Within a single day, the same visitor therefore sees the same variant (XICTRAQ documentation). On the next day a new session is created and assigned again at random. The same applies when the connection changes, for example from mobile data to home Wi-Fi, and to visitors who stay in the shop past the change of day.

In an ordinary week this often matters less, if fewer purchases are prepared over several days. During promotion week the pattern is an obvious one: compare on Monday, buy on Friday. Many online shoppers also use several devices in general: 73 percent of them shop on a smartphone and 52 percent on a notebook (Bitkom 2025), together well over 100 percent. For the test, this creates four effects:

  • The purchase counts for the variant of the buying session. If variant B persuaded on Monday and the Friday session was assigned variant A, the conversion counts for A.
  • The persuading session counts as a visit without a purchase. The Monday session with variant B appears in the report without a conversion.
  • Purchases outside the test page drop out. If the buying session does not load the page with the experiment at all, for example because the visitor goes straight from a wish list to checkout, the purchase counts for no variant.
  • The same person enters several times. Each new daily session is a new entry in the population, even if the same person is behind it.

A simple calculation shows the order of magnitude. Suppose variant B really raises the purchase probability by 10 percent. If the decision for 40 percent of purchases is made in an earlier session, those purchases spread randomly across both variants and add nothing to the difference. What remains measurable is roughly 6 percent uplift (0.6 × 10). Because the required sample is roughly inversely proportional to the square of the difference you are looking for, the test then needs about 2.8 times as many sessions (1 / 0.6²). All values in this calculation are assumptions. The direction is typical: repeat visits pull the measured effect toward zero. The sample size calculator shows what a smaller minimum effect means for the number of sessions you need.

The limit of cookieless assignment

A/B assignment in XICTRAQ® deliberately does not recognize visitors across days. That is what makes testing without storing anything on the device possible, and it is also the reason for the dilution. Because of it, the measured difference tends to be smaller than the effect on the individual purchase decision. A small or missing difference during this week is thus no proof that a variant has no effect.

Pause, extend or read as a seasonal segment

For every test that runs into the promotion period, a deliberate decision before the promotion starts pays off. Which option fits depends on what the result will be used for later. The following overview sorts the most common situations.

SituationRecommendationIn XICTRAQ®
Test has been running for weeks, the result is meant for the whole yearPause it for the promotion period or leave the promotion days out of the analysisStatus "paused", then resume; set the analysis period to the weeks before and after
Test only concerns promotion content such as a banner or landing pageA separate experiment with its own name that runs only during the promotion periodCreate a new experiment and end it after the promotion
Test keeps running, the result should show both situationsRead the promotion period as a separate segment instead of mixing it with normal weeksChoose the analysis period per window and put the intervals side by side
Planned sample size not yet reached before the promotionDo not top it up with promotion days; resume or restart after the seasonRecalculate the sample with a smaller minimum effect
Variant changes price display or discount presentationOnly test if both variants meet price indication rulesUse the result only for weeks with discounts

Pausing and ending differ considerably in XICTRAQ®. Paused and ended tests no longer assign new visitors (XICTRAQ documentation). With a paused test, the script on the page receives no variant, so the page shows what it shows without a variant, usually the original. Conversions from sessions assigned before the pause still count. An ended test, by contrast, is closed: it cannot be resumed, and conversions after the end no longer count. If you want to keep testing after the promotion, pause the test instead of ending it.

Extending in this situation does not mean letting the test run until Christmas. December is also a season, with gift purchases, delivery deadlines and holidays. Weeks that match a shop’s normal operations often only come around again in the new year, depending on the product range. If you need a test for the promotion, set it up as a separate experiment and name it so the period is part of the name:

promo-banner-test.js
// Separate experiment for the promotion period only
xictraq.experiment('promo-banner-bw-2026').then(variant => {
  // without a running test no variant is returned: the original stays
  if (variant === 'B') {
    document.querySelector('.promo-banner')?.classList.add('promo-banner--b');
  }
});

// on the order confirmation page
xictraq.convert('promo-banner-bw-2026');

The name records in the report what the result stands for. Nobody will mistake a test called "promo-banner-bw-2026" for a year-round result. Once the promotion is over, it is ended, and a new test with its own sample size planning starts for normal weeks if needed.

Analyze normal weeks and the promotion period separately

If a test runs straight through the promotion period, it can still be read cleanly. In the A/B report of XICTRAQ®, the table follows the selected date range: it counts the assignments within the range and their conversions up to its end. Each variant shows visitors, conversions, conversion rate, uplift with interval and p-value. This turns the promotion period into a separate segment that can be placed next to the weeks before and after. Here is how to proceed:

  1. Note the shop’s promotion period with dates, for example November 20 to 30, including early offers for newsletter subscribers.
  2. As the first window, select the normal weeks before the promotion and note the uplift interval.
  3. As the second window, select exactly the promotion period and write the interval next to it.
  4. After the promotion, analyze a third window once enough sessions have accumulated.
  5. Look at the whole interval for each window, not just the winner label.
  6. When reading several windows, apply a stricter threshold to the p-value.

The chart above shows what such a picture can look like. In the normal weeks, the uplift of B is +13 percent with an interval from +3 to +23 percent; in the promotion period it is −2 percent with an interval from −11 to +7 percent. These are example values. Read them like this: in the normal weeks the entire interval lies above zero, so an advantage of B is proven there. In the promotion period the interval includes zero, so no advantage is proven there. The fact that the two intervals overlap means neither that the effect is the same in both periods nor that it differs. For the decision, the separate reading is usually enough: adopt B for normal weeks and treat promotion week as a case of its own.

Reading several windows means making several comparisons, and with each one the chance rises that one of them stands out by accident. The Bonferroni method allows multiple comparisons and intervals while maintaining an overall confidence level (NIST/SEMATECH e-Handbook). To do so, each individual comparison is set more strictly: with three windows and an overall error rate of 5 percent, the threshold per window is about 0.017 (0.05 / 3). In XICTRAQ®, a variant is marked green as the winner once the p-value drops below 0.05 and the uplift is positive (XICTRAQ documentation). If you read three windows, check the displayed p-value against the stricter threshold yourself.

A segment is a finding about that segment

Experimental design has a term for this: the block. It serves to isolate a systematic effect so that it does not obscure the main effects (NIST/SEMATECH e-Handbook). The promotion period is such a block. A result from it applies to weeks with a promotion, a result from the normal weeks to weeks without one. Reporting both separately loses no information and avoids the wrong transfer.

What belongs in the report after Black Week

A test report after promotion week is useful if it is still understood next year. That takes more than the winning variant. Costs and revenue per campaign belong in the ROAS report for Black Week, and the comparison with the previous year needs the calendar shift of Black Friday. For the experiment itself, a few details are enough:

  • Name of the experiment, variants and exact runtime with dates
  • The shop’s promotion period and which discounts ran during that time
  • Per period: sessions, conversions, uplift with interval and p-value
  • A note that assignment is per daily session and returning visitors count several times
  • Technical anomalies during the test period, such as longer load times under load
  • Decision with scope: adopted for promotions, adopted for normal weeks, or to be retested

Before a result goes into the report as a design effect, it is worth checking technical causes. A button that does not respond under load shows up in rage and dead clicks, a checkout where one payment method fails shows up in the payment method analysis. Whether the site was reachable throughout promotion week at all is shown by uptime monitoring. A variant that loses only because its image does not load under load is not a design finding.

Decide on your tests before the promotion starts

Only a few days lie between the publication of this article and Black Friday. Go through every running test once and decide: pause it, run it as a separate promotion experiment, or let it continue and read it as a segment. Note the shop’s promotion period with dates before the first discount starts, so the windows fit exactly later. And for new tests, plan for a smaller measurable effect if many of your customers plan over several days.

A/B testing is part of cookieless analytics and available as a premium module from the Business plan (XICTRAQ documentation); the plans are listed on the pricing page. The documentation on A/B tests and insights describes assignment, status and analysis in detail, and the live demo lets you look at the report without signing up. Reading the promotion period as a separate segment keeps both: the finding for Black Week and a clean baseline for the weeks after, measured with analytics that needs no banner.

Sources and studies

This article is based on: NIST/SEMATECH e-Handbook of Statistical Methods, sections 7.3.3 (comparing two proportions), 5.7 (glossary of experimental design: interaction, randomization, blocking) and 7.4.7.3 (Bonferroni method); Eurostat, glossary "Seasonal adjustment"; Destatis, glossary "Kalender- und saisonbereinigte Werte"; Bitkom, press release "Black Friday: Die Hälfte geht auf Schnäppchenjagd" of November 24, 2025, representative survey by Bitkom Research among 1,120 internet users aged 16 and over in Germany, calendar weeks 40 to 41 of 2025; bevh, press releases of December 10, 2024 and December 11, 2025 based on the consumer survey "Interaktiver Handel in Deutschland" with 40,000 respondents aged 14 and over per year; German Price Indication Ordinance (PAngV), Section 11; XICTRAQ documentation on A/B tests and insights; XICTRAQ feature page Cookieless Analytics. The curves, intervals and the dilution calculation are based on assumed values.

Can I transfer an A/B result from Black Week to the whole year?
Usually not without another test. The result applies to the visitors and conditions of promotion week: more planned purchases, discounts, higher load. The effect of a variant can depend on these. It makes sense to treat the promotion result as a hypothesis and to test the variant again in normal weeks.
Should I pause a running test during promotion week?
If the result is meant for normal weeks, pausing is usually the simplest route. In XICTRAQ®, a paused test assigns no new visitors, and the page then typically shows the original. Alternatively the test keeps running and you analyze the promotion days as a separate window. You should not end it, because an ended test cannot be resumed.
Why can a returning visitor see a different variant?
Assignment is tied to a cookieless session whose key changes daily. Within a single day the same visitor sees the same variant; on another day or after a change of connection they may be assigned again. That is the price of storing nothing on the device, and it dilutes the measured difference, especially in weeks with many planned repeat visits.
How do I read promotion week as a separate segment?
In the A/B report, select the promotion period as the date range. The report then counts the assignments within that range and their conversions up to its end. Place the result next to a window from normal weeks and compare the uplift intervals, not just the winner label.
What does it mean when the intervals of two periods overlap?
An overlap initially only shows that the data are compatible with several readings. It proves neither that the effect is the same nor that it differs; only a separate comparison of the two effects answers that. For each period on its own, what is informative is whether its interval includes zero: if it does, no advantage of the variant is proven there.
May I test price or discount displays during promotion week?
In principle yes, provided both variants meet the price indication rules, in particular stating the lowest price of the last 30 days when announcing a price reduction. The result of such a test typically applies only to weeks with discounts. If in doubt, clarify the legal side before the start.

Related in XICTRAQ®

Start in 60 seconds

Let XICTRAQ® watch your website.

Free plan with no fixed term available. No credit card, no installation, no hidden fees.