Bots Are the Majority of HTML Traffic in 2026
The shift is historic. For years the bot share of web traffic hovered around the 40-to-50-percent mark, but in 2026 the ratio tipped: 57.5% of HTML requests on the network came from automated programs, only 42.5% from humans (Cloudflare Radar). The measurement covers exactly the HTML documents that also carry analytics scripts and advertising - not video streaming, email or gaming. The finding therefore hits precisely the pages on which reach and conversion are measured.
Within this bot share, AI crawlers are growing fastest. The vast majority of their activity serves training: over a twelve-month period, 80% of AI crawling went to collecting training data, 18% to search and only around 2% to user-triggered actions (Cloudflare Radar). The training share rose year over year from 72% to 79% (Cloudflare Radar) - so the crawlers are fetching increasingly more content for models, not for directly answering a user question. In the first week of August 2025, ClaudeBot and GPTBot together accounted for nearly half of observed AI crawling activity (Cloudflare Radar).
Every AI crawler that slips into analytics as a visitor dilutes sessions, dwell time and conversion rate - just like classic bots. Because AI crawlers preferentially fetch content-rich pages, the distortion is not even but directed. How to sort out automated traffic server-side without losing genuine users is covered in the article on how to filter bot traffic cleanly from your analytics.
It is important to read these shares not as a snapshot but as a trend. The bot share rises year after year, and AI crawlers drive that development. A website still working with unfiltered raw traffic today measures an increasingly distorted picture every quarter - while carrying a growing technical load that nobody ordered.
GPTBot, ClaudeBot and Co: Who Is Actually Crawling
To measure and steer AI crawlers, you first have to know which machines you are dealing with. The major providers do not run a single crawler but several - each with its own purpose and its own identifier. This very separation is later the key to targeted steering.
GPTBot collects content for model training (user agent GPTBot/1.4), OAI-SearchBot supplies pages for ChatGPT search, and ChatGPT-User fetches pages in response to a direct user action. GPTBot and OAI-SearchBot obey robots.txt and can be set independently of each other (OpenAI GPTBot documentation).
ClaudeBot fetches web content for training, Claude-User retrieves pages on behalf of individual users, and Claude-SearchBot improves search results. All three respect robots.txt and support the Crawl-delay directive (Anthropic ClaudeBot documentation).
Googlebot crawls for classic Google Search, while Google-Extended solely controls whether content is used for Gemini and Vertex AI. Google-Extended has no effect on search ranking and can be governed separately via its own robots.txt token (Google Search Central).
The crucial insight: training crawlers and search crawlers are separate identifiers at all three providers. GPTBot is not OAI-SearchBot, ClaudeBot is not Claude-SearchBot, and Google-Extended is not Googlebot. This lets you keep training out without sacrificing visibility in AI-assisted search answers. The prerequisite is a cleanly maintained robots.txt - the pitfalls lurking there are covered in the article on how to avoid robots.txt mistakes.
GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Googlebot obey robots.txt (OpenAI GPTBot documentation; Anthropic ClaudeBot documentation; Google Search Central). For user-triggered fetches such as ChatGPT-User, OpenAI explicitly notes that robots.txt rules may not apply, because the request stems directly from a user action (OpenAI GPTBot documentation). For reliable detection the identifier alone is therefore not enough - the origin has to be verified as well.
Why AI Crawlers Generate Measurable Server Load
AI crawlers are not only a measurement problem but a real load on infrastructure. Unlike a human reading a few pages, crawlers work systematically: they fetch large volumes of subpages in bulk, often including rarely visited depths of a site. That access is expensive precisely because it bypasses the caches and reaches all the way into the core data centres instead of being served from the cheap intermediate store.
The Wikimedia Foundation has quantified the scale. Since January 2024, bandwidth for downloading multimedia content rose by 50% - driven mainly by automated programs scraping the image catalogue for AI models (Wikimedia Foundation). Even more telling is the load distribution: at least 65% of the most resource-intensive traffic comes from bots, although bots make up only around 35% of pageviews (Wikimedia Foundation). A small share of requests therefore causes a disproportionately large share of the cost. The Foundation now blocks or throttles about 30% of the automated requests that do not follow the rules - amid up to billions of requests per day (Wikimedia Foundation).
More than half of AI bots' crawl traffic goes to re-fetching unchanged pages (Cloudflare Radar). That means a large part of the load produces no new value at all but simply repeats what the crawler already knows. For the site operator this is doubly annoying - paying compute and bandwidth for fetches that benefit neither them nor the crawler.
For smaller sites without the reserves of a large foundation, the consequence is tangible: an aggressive crawler fetching hundreds of pages in a short time can visibly push response times up and, in extreme cases, take a server to its limits. Anyone who watches response times not only on average but at the outliers spots such spikes early - how that works with percentiles is explained in the article on response-time percentiles P95 and P99.
Separating AI Crawlers From Real Visitors: Logs and Traffic
To cleanly exclude AI crawlers from reach, you need reliable detection. Three signals work together, and none is sufficient on its own.
- User-agent identifier - GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot and Googlebot declare themselves in the identifier. That is the first, fastest filter, but also the easily forged one: a script can fake any identifier.
- IP verification - OpenAI publishes the IP ranges of its crawlers in publicly retrievable lists (OpenAI GPTBot documentation), and Anthropic provides a list through which a request can be confirmed as a genuine Anthropic bot (Anthropic ClaudeBot documentation). Only a request that sends both the matching identifier and an IP from the official range is truly the stated crawler.
- Network origin (ASN) - requests from data-centre networks are highly likely to be machine-generated, because humans practically do not browse from a hosting network. This signal also catches disguised automation that sends an inconspicuous browser identifier.
In server logs, AI crawlers reveal themselves through typical patterns: they systematically pull the sitemap, follow links wide and deep, often keep fixed time intervals and carry the characteristic identifier. For analytics, the decisive point is that these requests are never counted as a session in the first place. A server-side, cookieless analytics checks the network origin on entry and flags data-centre traffic before it forms a session - regardless of the identifier it claims.
| Detection path | What it delivers | Limit |
|---|---|---|
| User-agent filter | Catches openly declared crawlers (GPTBot, ClaudeBot) at once | Identifier is forgeable; disguised bots slip through |
| IP verification | Confirms genuine crawler via official IP list | Only for providers with published ranges |
| ASN / data-centre detection | Detects machine traffic by network origin | Needs a curated directory to avoid hitting end users |
| Combination of all three | Separates human and machine reliably | Ongoing maintenance required |
The effort pays off twice: clean analytics data and a clear picture of which crawler generates how much load. Only once AI crawlers are recorded separately can you decide which to welcome and which to throttle.
Steer via robots.txt and llms.txt Without Losing Visibility
The most common wrong reaction to AI crawlers is blanket blocking - and that can get expensive. Anyone who blocks all AI bots loses not only training crawlers but also the search crawlers that make a site visible in AI-assisted search answers. The solution lies in steering separately via the identifiers.
# Block training, keep search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Deliberately allow search crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /With this split a website stays discoverable for AI search while the pure training crawlers remain outside. One important detail: at OpenAI it can take around 24 hours for a changed robots.txt to take effect in its systems (OpenAI GPTBot documentation). At Google, a Disallow rule prevents crawling but not indexing - anyone who wants to keep a URL reliably out of the index needs noindex rather than robots.txt alone (Google Search Central).
Alongside robots.txt, a second, complementary approach has emerged in llms.txt. It is a proposed convention: a Markdown file in the root directory that provides language models with curated, well-structured content and references. The distinction matters - llms.txt is an offer for content preparation, not an access control. Access control is handled by robots.txt; content curation for models can be handled by llms.txt. Anyone who wants to be present in AI answers combines the two: allow search crawlers and prepare their own content cleanly for models. How to measure presence in AI systems is shown by AI visibility.
Blocking all AI crawlers outright does reduce load but can cost discoverability in AI search engines. Conversely, free crawling means full server load. The workable line lies in between: limit training in a targeted way, allow search, throttle aggressive access and continuously measure the actual load. That keeps the decision data-driven rather than reflexive.
Anyone securing the scripts and third-party access on their site anyway should see AI crawler steering as part of broader web hygiene. The current attack classes are summarised in the article on the OWASP Top 10 2025; how to secure client-side scripts in the payment context is covered in the article on PCI DSS 4.0.1 client-side security.
Spot Load Spikes and Response-Time Anomalies Early
Steering via robots.txt only works for crawlers that observe it. For the rest - and for the case where an otherwise well-behaved crawler suddenly turns aggressive - observation is needed. A sudden wave of hundreds of fetches in a short time shows up first in two places: in rising response times and in an atypical traffic pattern.
This is exactly where continuous uptime monitoring comes in. If a site is queried regularly from several regions, a crawler-induced slowdown stands out long before real visitors experience it as a sluggish page. Instead of rigid thresholds, an adaptive baseline that knows the normal daily and weekly rhythm helps, alerting only when the response time truly falls outside the range. How such automatic anomaly detection works is described in the article on AI-assisted anomaly detection in website monitoring.
When response times suddenly rise, an aggressive crawler is one of the first causes to check. Multi-region checks separate a genuine load problem from a regional network fault.
An alert the moment response time exceeds the usual range buys a head start - by email, Slack, Discord or Telegram, before the site becomes noticeably slow for users.
An atypical rise in requests, especially from data-centre networks, is an early indicator. Separately recorded crawler access is what makes such patterns visible in the first place.
Knowing the load lets you react deliberately: set a crawl-delay, throttle individual crawlers or add capacity - a conscious decision rather than a sudden outage.
What XICTRAQ Delivers for AI Crawler Monitoring
XICTRAQ combines the three layers that make AI crawler traffic manageable: clean measurement, ongoing monitoring of the load and the classification of suspicious access. The following overview shows which building blocks work together.
Availability and response times are measured from several regions, so crawler-induced load spikes stand out before the site slows down - the core of uptime and response-time monitoring.
Data-centre traffic is flagged by ASN and removed from reach, so AI crawlers do not inflate analytics - the basis of cookieless measurement.
Unusual response times or traffic patterns trigger an alert by email, Slack, Discord or Telegram - based on an adaptive baseline rather than rigid thresholds.
A real browser checks which scripts and third-party calls a page loads - part of the security scan that makes suspicious access visible.
Load trends and availability can be prepared in clear reports and a public status page - see reporting and status page.
While the load is being steered, presence in AI answers stays measurable, so the balance between protection and discoverability remains data-driven.
The common denominator is the real visitor's perspective: AI crawlers are detected, their load is observed, and reach stays free of automated noise. The practical entry points are the pricing overview and a no-obligation live demo, in which response times and traffic patterns can be traced directly.
This article draws on data and documentation from: Cloudflare Radar (57.5% bot share of HTML requests versus 42.5% humans in June 2026; 80% of AI crawling for training, 18% search, 2% user actions, training share up from 72% to 79% year over year; ClaudeBot and GPTBot together nearly half of AI crawling activity in the first week of August 2025; crawl-to-visit ratios in July 2025 of roughly 38,000:1 for ClaudeBot, 1,091:1 for GPTBot and about 5:1 for Googlebot; more than half of AI crawl traffic re-fetching unchanged pages), Wikimedia Foundation (multimedia bandwidth up 50% since January 2024; at least 65% of the most resource-intensive traffic from bots at around 35% of pageviews; throttling or blocking of about 30% of non-compliant automated requests), OpenAI GPTBot documentation (identifiers GPTBot, OAI-SearchBot, ChatGPT-User; robots.txt behaviour; published IP ranges; around 24 hours for robots.txt changes to take effect), Anthropic ClaudeBot documentation (identifiers ClaudeBot, Claude-User, Claude-SearchBot; robots.txt and Crawl-delay; IP list for verification) and Google Search Central (Googlebot and Google-Extended; separate robots.txt token; difference between crawling and indexing). Information without warranty; legal and technical decisions should be assessed case by case.