Why Bot Traffic Destroys Data Quality
Web analytics is only as good as the data that flows into it. As soon as a meaningful share of measured requests comes not from people but from machines, every metric shifts - and not evenly, but selectively. A bot that loads a page and moves straight on creates a session with zero seconds dwell time and a 100 percent bounce rate. A scraper that systematically works through hundreds of subpages artificially inflates pages per session. Both appear in the dashboard as visitors, even though neither will ever buy a product, complete a form or subscribe to a newsletter.
The distortion hits three central metrics at once. First, reach: session and user counts run too high because every crawler is counted. Second, engagement: dwell time, scroll depth and pages per session are either dragged down (by single-click bots) or driven up (by systematic scrapers). Third - and economically the most painful - the conversion rate: it is the ratio of completions to sessions. Because bots inflate the denominator without ever converting, the conversion rate looks lower than it actually is. Anyone who then tweaks the landing page, price or checkout is optimising against a measurement artefact.
Bot traffic is not evenly distributed across channels. Scrapers and AI crawlers hit content-rich pages first, data-center bots arrive via direct visits rather than paid campaigns, and automated requests cluster at particular times of day. This shifts not only the totals but the distributions - the apparent channel mix, the supposed best time of day, and the conversion rate per source. Decisions on such data correct around actual user behaviour.
What makes it especially treacherous is that the errors propagate. A/B tests count bot sessions into both variants and dilute statistical power. Attribution models spread revenue across channels that in truth attracted a disproportionate share of bots. And automated bidding strategies learn from corrupted signals. Cleaning at the source - sorting out non-human requests before they enter the analysis - is therefore not a cosmetic touch-up but the precondition for every downstream analysis to remain reliable.
About Half of Web Traffic Is Automated
The data is unambiguous and reached a turning point in 2025. For the first time in a decade, automated traffic surpassed human activity, making up 51% of all web traffic while humans contributed only 49% (Imperva Bad Bot Report). The share of clearly malicious bots - scrapers, credential-stuffing scripts, scalpers and fraud automation - rose to 37% of internet traffic, up from 32% the year before (Imperva Bad Bot Report). The remaining machines are so-called good bots such as search-engine crawlers that declare their purpose openly, yet are still not human visitors and have no place in a reach measurement.
A second, independent measurement confirms the picture from a different angle. On HTML pages - precisely the documents that also carry analytics scripts - only 47% of requests came from humans in early December 2025, while non-AI bots generated 44% (Cloudflare Radar 2025 Year in Review). Specialised AI bots contributed an average of 4.2% of HTML requests over the year, with Googlebot alone at 4.5% (Cloudflare Radar 2025 Year in Review). The trend points steeply upward: so-called user-action crawling, where machines fetch pages on behalf of individual users, grew more than 15-fold in 2025 (Cloudflare Radar 2025 Year in Review).
- Data-center bots - requests from the networks of AWS, Google Cloud, Azure and hosting providers; human users practically never browse from such networks.
- AI crawlers - bots that collect content for training and answer systems; on average 4.2% of HTML requests (Cloudflare Radar 2025 Year in Review).
- Headless browsers - full but remotely controlled browsers that send a genuine browser identity and are hard to tell apart from a human.
- Residential-proxy bots - attackers routing their traffic through consumer connections; 21% of bot attacks use such proxies to blend in with real users (Imperva Bad Bot Report).
- Good bots - search-engine and monitoring crawlers; legitimate, yet still not human visitors and therefore to be removed from reach.
The key insight is that these shares do not stay stable but rise year after year. An analytics configuration that catches a useful portion of bots today steadily loses discriminating power without maintenance. The same structural pattern - growing measurement gaps from technology that evolves faster than the standard tools - is described in the article on ad blocker data loss and first-party measurement: there too, it is not a one-off setup but ongoing adjustment that determines data quality.
Why Standard Filter Lists Catch Only Part of It
Most analytics tools filter bots using lists of known signatures. The industry standard for this is the IAB/ABC International Spiders and Bots List, maintained by the Alliance for Audited Media and updated monthly (IAB/ABC International Spiders and Bots List). It works in two stages: first an allowlist checks whether the browser identity matches a known, valid pattern, then a denylist removes any remaining signatures recognised as bots (IAB/ABC International Spiders and Bots List). This dual-pass method supports the Media Rating Council's General Invalid Traffic Detection and Filtration Standard and is a solid basis - but it has a fundamental limit.
Invalid traffic is generated by bots, click farms and other non-human activity. It inflates website and campaign figures and distorts the measured metrics, making reach and origin hard to determine reliably.
— IAB/ABC International Spiders and Bots List
The limit lies in the method itself: signature lists recognise a bot by what it claims about itself - above all by its browser identity. And that is exactly what the modern, commercially motivated bots have learned to fake. Three categories therefore slip through the standard filters systematically.
Remotely controlled but complete browsers run JavaScript and send an entirely inconspicuous browser identity. To a signature-based list they look like a human on current Chrome - the single most impersonated browser, accounting for 46% of all faked identities (Imperva Bad Bot Report).
New bots for AI training and answer systems appear faster than lists can absorb them. Some declare themselves honestly, many do not - and even the honest ones, at 4.2% of HTML requests, belong out of the human reach figure (Cloudflare Radar 2025 Year in Review).
21% of bot attacks run through the home connections of real internet customers (Imperva Bad Bot Report). The traffic then appears with an entirely ordinary consumer IP and a legitimate browser identity - together this makes detection by user-agent alone nearly hopeless.
The shared problem: a list can only catalogue what is already known. Once a bot sends a credible browser identity and comes from an inconspicuous network, the signature offers no leverage. This is exactly where consent-free, server-side measurement comes in, whose basic mechanics the article on cookieless tracking without consent explains in detail. The way out is to stop asking what a request claims about itself and instead ask where it technically actually comes from.
Filtering Server-Side by ASN and Data-Center Detection
The most effective lever against disguised bots is a property that is hard to fake: the origin of the network. Every IP address belongs to an autonomous system (AS) that is uniquely assigned to an operator via an ASN number - an access provider, a mobile network, or indeed a data center. A human visiting a website comes over the connection of an access or mobile provider. A request originating from the network of AWS, Google Cloud, Azure or a bulk host, by contrast, is practically never a private surfer but a server - and thus with high probability a bot.
Server-side filtering uses exactly this signal. Instead of trusting the user-agent, the visitor IP is checked on entry against a directory of ASN and data-center ranges. If the IP falls into a hosting range, the request is marked as machine and removed from the reach measurement before it even forms a session. The decisive advantage over signature lists: the bot can fake its browser identity as often as it likes - that does not change its network origin. A scraper on a cloud instance remains a request from the data center, whatever browser it pretends to use.
ASN-based filtering is only clean when it stays precise. Mobile providers, cable and DSL operators, or business fibre must not be classified as a data center, otherwise real visitors vanish from the statistics. What matters is therefore a carefully curated ASN directory that captures hosting and VPN networks without hitting consumer networks. XICTRAQ separates data-center ASNs from consumer and carrier networks and discards only the data-center requests - here an overly broad filter would be more harmful than none at all.
| Property | Signature-based filter list | Server-side ASN detection |
|---|---|---|
| Detection feature | browser identity (user-agent) | network origin (ASN / data center) |
| Faked user-agent | is missed | stays ineffective |
| Headless browser | appears as human | recognisable as hosting request |
| Residential proxy | barely detectable | partly detectable by network |
| Maintenance effort | monthly list updates | continuously maintained ASN directory |
| Risk to real visitors | low | low with a curated directory |
The second building block is the first-party architecture. When measurement is served via your own domain and evaluated server-side, the network origin of each request can be determined right at the entry point - before any data is aggregated. This is also the basis for keeping the analysis data-minimal and free of transfer to third parties. How conversions can be captured in full despite blockers and disguise is explored in the article on conversion tracking without third-party scripts; what a privacy-friendly analytics basis looks like overall is shown in the comparison with a GDPR-compliant analytics solution.
- Network origin instead of self-declaration - every IP is checked against an ASN and data-center directory, not against its browser identity.
- Curated ASN directory - hosting and VPN networks are captured, consumer and mobile networks deliberately excluded.
- Filtering at the entry point - machine requests are marked before they form a session, rather than correcting figures afterwards.
- First-party evaluation - origin is determined on your own domain, without passing data to third parties.
- Ongoing maintenance - the directory is continuously updated because new hosting ranges and bot networks appear constantly.
Chasing bots by user-agent means running after the forgers. Recognising them by network origin turns the tables: a data center stays a data center, regardless of which browser the request pretends to use. This turns an inflated raw statistic into a reliable data basis - with real sessions, genuine dwell time and a conversion rate that rests on actual visitors rather than machines.
What XICTRAQ Delivers for Clean Analytics Data
At XICTRAQ, bot filtering is not an add-on switch but built into the measurement. Every request is checked against its network origin on entry, so data-center traffic is not counted as a visitor in the first place. The overview below shows which building blocks work together so that the analysis rests on real users rather than machines.
Requests from hosting networks such as AWS, Google Cloud or Azure are marked by ASN and removed from reach - without hitting mobile and consumer networks. The basis of cookieless analytics.
A continuously maintained directory distinguishes hosting and VPN networks from consumer and carrier networks. The same geo and ASN data basis is available for your own analyses via the Geo API.
Sessions, dwell time, bounce rate and conversion rate are calculated without bot requests - the dashboard figures reflect real user behaviour rather than a mix of human and machine.
A real browser checks which scripts and tags a page loads and which external requests occur - part of the security scan that also surfaces suspicious third-party calls.
Uptime, response times and Web Vitals are monitored from several regions so that drops in reach are not wrongly attributed to bots. More at uptime monitoring.
Sudden traffic spikes from data centers or unusual patterns trigger a notification by email, Slack, Discord or Telegram - before they turn into a wrong analysis.
The common denominator is measurement from the perspective of the real visitor and the consistent separation of human and machine by network origin. Anyone who also wants to secure the technical resilience of their infrastructure more broadly will find related topics in the article on monitoring quantum-safe TLS encryption and on secured email encryption with MTA-STS and TLS-RPT.
Establishing Bot Filtering as the Standard for 2026
The numbers leave little room for interpretation: when automated traffic already makes up 51% of all requests (Imperva Bad Bot Report) and only 47% of HTML-page requests come from humans (Cloudflare Radar 2025 Year in Review), unfiltered raw traffic is no longer a usable basis for decisions. Anyone planning marketing budgets, product decisions and A/B tests on a mix of human and machine is optimising against a distorted picture - and the share of distortion grows every year.
The switch is not a major project but an architecture decision: server-side filtering by network origin, a carefully curated ASN directory, and first-party measurement that sorts out bots before they enter the analysis. Combined with ongoing maintenance, data quality stays stable even as new bot networks and AI crawlers arrive. The practical entry point runs via the pricing overview and a no-obligation live demo, where the difference between raw and cleaned figures can be seen directly.
This article draws on data from: Imperva Bad Bot Report (51% automated traffic in 2025, 49% human, 37% malicious bots versus 32% the prior year, 21% of attacks via residential proxies, 46% impersonated Chrome user-agent), Cloudflare Radar 2025 Year in Review (47% human and 44% non-AI-bot share of HTML requests in early December 2025, 4.2% AI bots, 4.5% Googlebot, more than 15-fold growth in user-action crawling) and the IAB/ABC International Spiders and Bots List (monthly maintained signature list, dual-pass method, Media Rating Council's General Invalid Traffic Detection and Filtration Standard). Legal statements do not replace legal advice.