__usage_ai-training in the User-Agent line, described in an individual draft without formal standing in the standards process (IETF Datatracker). This article shows what the token covers, what works in the meantime, how to add the group without hitting search crawlers, and how to keep an eye on the file after every deploy.What was announced on October 30
The announcement is short and its key sentences are clear. Website owners are to be able to express their AI training preference for several AI providers in one step instead of maintaining separate instructions for every developer (Google Search Central). If a site uses the new token to opt out of training and does not explicitly name Google-Extended, the announcing provider plans to treat it as if Google-Extended had been disallowed. Google-Extended is that provider's own training token, offered since 2023 (Google Search Central).
Today, we're announcing our plans to support a shared AI training control in robots.txt.
— Google Search Central Blog, "Supporting a shared AI training control for websites", October 30, 2026
Two points matter for the assessment. First, the existing provider-specific tokens remain in place and keep working alongside the shared signal. Second, this is an intention with a timeline, not a fixed date: the provider aims to support the control together with other AI providers in 2027 and to report on progress (Google Search Central). The same provider's crawler documentation says early 2027 (Google Crawling Infrastructure).
The token comes from the Internet-Draft draft-kelley-aipref-training-preference-signal-00. According to the Datatracker, it is an individual draft that is not endorsed by the IETF and has no formal standing in the standards process (IETF Datatracker). Internet-Drafts are valid for a maximum of six months and may be updated, replaced or withdrawn at any time; this one expires on May 3, 2027 (IETF Datatracker). Token name, rules and timeline may therefore still change before support arrives.
How the policy token works
The draft introduces no new directive and no new grammar. A site operator writes an ordinary robots.txt group whose User-Agent line does not name a crawler but a policy token for a purpose of use (IETF Datatracker). Path language, longest-match precedence and group structure stay as RFC 9309 defines them for the Robots Exclusion Protocol. Parsers that do not know the token skip the group like that of an unknown crawler. The example from the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
The group is read like any other: where a path is disallowed, the operator has expressed a preference against AI training; where it is allowed, a preference in favor. In the example, that means no for the whole website and yes for the press material under /press-kit/. If there is no group with the token at all, no preference has been expressed (IETF Datatracker, Section 4.2). Four rules from the draft matter most in practice:
- The group for all crawlers does not count. A group with
User-agent: *must explicitly not be treated as a match for the token (IETF Datatracker, Section 4.2). A blanketDisallow: /for all crawlers therefore expresses no training preference, but it does block search. - Case does not matter, spelling does. The token is matched like any product token, case-insensitively but otherwise exactly (IETF Datatracker, Section 4.1).
__usage_ai_trainingwith an underscore instead of a hyphen is a different token and expresses nothing. - Not a crawling instruction. A provider may use the preference to avoid fetches, but does not have to (IETF Datatracker, Section 4.4).
- Independent of crawler rules. Every crawler keeps reading its own group as before. Content usage and crawl control live in the same file but are separate from each other (IETF Datatracker, Section 4.3).
The draft does not define "AI training" itself. It takes the definition from the working group's vocabulary draft: using an asset to modify the learned parameters of a generative AI model (IETF aipref working group, vocabulary draft). Training models that perform exclusively non-generative tasks is not included. The vocabulary has a separate category for search; the token from the announcement does not touch it.
Training is not search: what the token leaves alone
The new signal is narrow: it covers exactly one purpose of use. Whether a page is crawled, indexed and shown in search results is still governed by the groups for the crawlers themselves, usually the User-agent: * group. For Google-Extended, the token onto which the announcing provider maps the new signal, the documentation states explicitly that it does not affect inclusion in web search and is not a ranking signal (Google Crawling Infrastructure).
One caveat still belongs on the table before you set the opt-out. According to the documentation, the same token controls not only the training of future model generations but also so-called grounding: content from the search index is passed to a model at prompt time so that answers in the provider's AI applications are more factual (Google Crawling Infrastructure). The announcement notes that websites which explicitly allow the token are eligible to appear in such answers and receive clicks (Google Search Central). If you opt out via the shared token without naming that token explicitly, expect this route into the provider's AI answers to disappear as well. That is a trade-off between protecting content and AI visibility, not a technical question. How presence in AI answers can be measured in the first place is covered in the article on measuring Generative Engine Optimization (GEO).
| Rule in robots.txt | Effect today | Effect after announced support |
|---|---|---|
| User-agent: * with Disallow: / | blocks crawling for all crawlers without their own group, including search | unchanged; explicitly does not count as a training preference |
| Group per training token, such as Google-Extended | works with the provider that documents the token | keeps working and takes precedence over the token at the announcing provider |
| Group __usage_ai-training with Disallow: / | skipped by current parsers as an unknown crawler | opt-out of training at participating providers, treated like a Google-Extended opt-out at the announcing provider |
What already works today: tokens per crawler
Until support actually arrives, only what providers document today has any effect: their own tokens per crawler or per purpose. Google-Extended has no separate user agent in requests and is read only in robots.txt as a control token (Google Crawling Infrastructure). Other providers separate training and search through different crawlers. OpenAI, for example, explains that disallowing GPTBot indicates the content should not be used for training generative AI foundation models, while a separate search crawler handles appearance in search results (OpenAI). A robots.txt that opts out of training with these two providers today while leaving search open looks like this:
# Search and indexing: group for all crawlers
User-agent: *
Disallow: /cart/
# AI training: one group per provider
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
This list grows with every provider and is only as complete as your last look at the documentation. This is exactly where the shared token comes in: it is meant to close the gap for providers you do not list individually. Which tokens are common and how to recognize them in the access log is covered in the article on llms.txt and AI crawlers; how to measure their load is covered in the article on tracking AI crawler traffic. You do not have to type the per-crawler groups by hand: the free robots.txt and llms.txt generator creates them for the common AI crawlers. It does not know the new token yet, so you add that group by hand.
According to the announcement, the provider-specific tokens continue to work alongside the shared signal (Google Search Central). The draft says specific expressions should take precedence over general ones and allows providers to treat their own tokens as more specific than the shared token (IETF Datatracker, Section 4.5). Existing blocks therefore stay, and the token is added as a fallback.
Which stance fits which website
Whether opting out of training makes sense depends on the business model. The token supports paths, so it can also say no for individual sections. Three typical starting points:
Product and category pages should be mentioned in AI answers. Rather than a blanket opt-out, an opt-out for costly sections such as guides, via paths in the token group, makes more sense.
The content is the product. An opt-out for the whole website with exceptions for press sections matches the example in the draft. Whether AI answers still link to the site belongs in the trade-off.
Every client website has its own robots.txt. Anyone adding the token for clients needs evidence per website of when the group went live, and a notification when it disappears.
Exceptions follow the familiar robots.txt logic: the rule with the longest match wins, and for equivalent rules Allow should take precedence (RFC 9309). An Allow: /press/ next to Disallow: / therefore works as expected. An Allow: /press without a trailing slash, by contrast, also covers a path such as /press-distribution-internal/, which is then released for training.
Adding the new group without hitting search crawlers
Because the token is just another group, its position in the file decides whether it works as intended. Under RFC 9309, a group consists of one or more User-Agent lines followed by rules; it only ends when another User-Agent line follows a rule (RFC 9309). This leads to the most common mistakes when adding it:
- The token line directly below another User-Agent line. If
User-Agent: __usage_ai-trainingsits immediately belowUser-agent: *, both form one group. ADisallow: /below then applies to every crawler without its own group, including search engines (RFC 9309). - A sitemap line as a supposed separator. Other records such as
Sitemap:do not terminate a group (RFC 9309). Google shows exactly this case: two User-Agent lines with a sitemap line between them count as one group, and the followingDisallow: /hits both crawlers (Google Crawling Infrastructure). - Blank lines do not separate. Under the RFC 9309 grammar, a blank line does not end a group either. Only a rule separates reliably: if the preceding group has at least one
AlloworDisallowline, the token line starts a new group.
User-agent: *
Disallow: /cart/
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
# Fallback for all other providers that read the token
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/
Sitemap: https://www.example.com/sitemap.xml
# Wrong: no rule between the User-Agent lines
User-agent: *
User-Agent: __usage_ai-training
Disallow: /
# Result: just one group, Disallow: / applies to every crawler
Two further limits: Under RFC 9309, crawlers must be able to parse at least 500 kibibytes (KiB) of a robots.txt (RFC 9309); Google parses up to 500 KiB and ignores whatever comes after (Google Crawling Infrastructure). In very large files, a group appended at the end may otherwise land beyond the limit. The file also has to be served as UTF-8 plain text; if the server returns an HTML page under /robots.txt, the parser tries to extract rules and ignores the rest (Google Crawling Infrastructure).
After adding the group, it pays to check the group structure before the file goes live. The free robots.txt and sitemap validator loads the file, splits it into User-Agent blocks with their rules and thus shows which lines belong together. General failure patterns, from an accidental Disallow: / to blocked resources, are covered in the article on common robots.txt mistakes.
When a change takes effect: cache, errors, redirects
A rule in robots.txt does not take effect the moment you deploy. Crawlers may cache the file; RFC 9309 says a cached version should not be used for more than 24 hours as long as the file is reachable (RFC 9309). Google generally caches the file for up to 24 hours, longer in case of timeouts or server errors (Google Crawling Infrastructure). A new rule may therefore take up to a day, and a mistake stays in force just as long.
More important is what a crawler sees when the file is missing or the server fails. Both typically happen during a deploy:
| Response to /robots.txt | Consequence under RFC 9309 | Consequence at Google | Consequence for the training rule |
|---|---|---|---|
| Client error (4xx), such as 404 after a build without the file | crawlers may access any resources | like a missing robots.txt, no crawl restrictions (except for 429) | disappears without replacement, as does every per-crawler group |
| Server or network error (5xx) | crawlers must assume complete disallow | no crawling for the first 12 hours, then the last good version for up to 30 days | the old version stays in force, search pauses |
| Redirect | follow at least five consecutive redirects | after five redirects treated like a missing file | only works if the target lies within the chain |
The details come from RFC 9309, Sections 2.3.1.2 to 2.3.1.4 (RFC 9309), and from Google's interpretation (Google Crawling Infrastructure). For the training rule, the first case is the tricky one. A deploy that does not ship robots.txt removes not only the new token but every per-crawler block, and no crawler will tell you about it. The second case hits search: if /robots.txt returns a server error, Google initially stops crawling (Google Crawling Infrastructure). How easily this happens in a release process is shown by the related pattern of a noindex leaking in a deploy.
Monitoring robots.txt after every deploy
A rule that only takes effect in 2027 is written today and has to survive every deploy, every content management system update and every change of host until then. XICTRAQ® SEO monitoring fetches the robots.txt of every monitored website on a schedule and compares it with the most recently stored version. When the content changes, the changelog gets an entry with severity warning that lists the added and removed lines together with their count, such as +3 for three added lines. That shows from which fetch onward the token group was in the file and whether a later deploy removed it again.
If the fetched robots.txt differs from the previous version, an entry appears with a date and usually with the added and removed lines. The schedulable SEO report includes changes to robots.txt.
If the group for all crawlers blocks the entire website, XICTRAQ® raises a critical alert. A separate token group does not trigger it, a merged one does.
An API check on /robots.txt verifies status 200 and whether the token line is in the text. If it fails twice in a row, the alert goes out through your configured channels.
The full-block alert works per group: it evaluates only the User-agent: * group and fires when that group contains Disallow: / without Allow: /. It therefore catches the case where the token line slips into the group for all, and stays quiet as long as the training group stands apart. It is delivered through the channels you have set up in notifications.
If the file disappears entirely, there is no new version to compare, and the line comparison stays silent. That is what the check for the token line is for: two assertions that must both pass; like the draft, the second ignores case:
[
{ "type": "status_code", "operator": "eq", "value": 200 },
{ "type": "text_contains", "value": "User-Agent: __usage_ai-training" }
]
For online stores, the announcement comes at an awkward time. Before Black Week, many campaign pages and tests go live, and each of these deploys can overwrite robots.txt. Add the training group separately from the campaign release and check the changelog entry before the campaign code follows. How to evaluate the tests from the same week properly is covered in the article on A/B tests in Black Week.
What remains open until 2027
The announcement describes a direction, not a finished standard. Four points are explicitly open:
- Name and rules of the token. The draft is in its first version and may be updated, replaced or withdrawn before support arrives (IETF Datatracker).
- Provider participation. It only takes effect with providers that actually read the token. The announcement speaks of support together with other AI providers but does not name any of them (Google Search Central).
- The rest of the vocabulary. The token uses only the definition of AI training; discussions on the remaining terms of the working group's vocabulary continue (Google Search Central).
- Enforcement. Technical enforcement of preferences is explicitly outside the working group's charter (IETF aipref working group, charter). RFC 9309 also makes clear that robots.txt rules are not a form of access authorization (RFC 9309). The token is a statement, not a block.
The aipref working group has a second draft, draft-ietf-aipref-attach, which attaches preferences through a separate Content-Usage field in robots.txt and in HTTP responses (IETF aipref working group, attach draft). That is a different mechanism that the October 30 announcement does not mention. A Content-Usage line does not count for the announced signal.
Legally, the technical signal is only one building block. How a reservation of rights for text and data mining (TDM) is worded so that it holds up under German copyright law is covered in the XICTRON® article on the TDM opt-out for online stores.
Roadmap: prepare now, switch over in 2027
- Define the goal. No training, but search: for many site owners, that is the starting point. Also clarify whether you want to appear in AI answers, because depending on the provider that hinges on the same token as training.
- Take stock. Which per-crawler groups are in robots.txt today, and which of them concern training? The free AI readiness check shows which known AI crawlers are allowed or blocked.
- Add the token group. As a group of its own after a group with rules, not directly below another User-Agent line, with path exceptions only where they are intended.
- Check the structure. Have the validator split the file into groups and verify that
User-agent: *has not gained a new block. - Ship it separately. As its own small deploy, not bundled with campaign or template changes.
- Monitor. Keep the changelog and the full-block alert active, plus the check for the token line.
- Follow up in 2027. Check the Datatracker for a new version of the draft and follow provider announcements; if the token name changes, adjust the group.
For the time being, the shared token replaces none of your existing rules. It is added as a fallback and takes effect in 2027 at the earliest, only with providers that read it. What it needs today is a robots.txt with cleanly separated groups and monitoring that makes changes visible.
This article is based on: Google Search Central Blog, "Supporting a shared AI training control for websites", October 30, 2026; Google Crawling Infrastructure, "List of Google's common crawlers" (Google-Extended section) and "How Google interprets the robots.txt specification"; IETF Datatracker, draft-kelley-aipref-training-preference-signal-00, individual draft dated October 30, 2026, valid until May 3, 2027; IETF aipref working group, vocabulary draft draft-ietf-aipref-vocab-08, attach draft draft-ietf-aipref-attach-05 and the working group charter; RFC 9309, Robots Exclusion Protocol; OpenAI, Overview of OpenAI Crawlers. Retrieved on November 2, 2026; product details as of November 2026.
What is the shared AI training signal in robots.txt?
__usage_ai-training in the User-Agent line. A Disallow expresses a preference against AI training per path, an Allow a preference in favor. It is based on an individual draft without formal standing in the standards process (IETF Datatracker).When does the token take effect?
Will I lose rankings if I opt out of AI training?
Should I delete my existing rules for individual AI crawlers?
Can the token accidentally block search engines too?
User-agent: * or is separated from it only by a sitemap or blank line, the groups merge, and a Disallow: / typically hits every crawler without its own group (RFC 9309).