Skip to content
GDPR compliant. Hosted in the EU.
14-day free trial →
robots.txtAI trainingIETF draft
SEO 17 min read 18 Nov 2026

Opting out of AI training via robots.txt: the shared signal

A shared token in robots.txt is meant to opt out of AI training across providers. What the draft covers, what applies until 2027 and how to keep search intact.

X

XICTRAQ® Team

XICTRAQ®

robots.txtAI trainingSEO-MonitoringAI visibility
2027
is when support for the shared token is planned (Google Search Central)
6 months
is the maximum validity of an Internet-Draft such as the token draft (IETF Datatracker)
24 h
is the longest a crawler should keep using a cached robots.txt (RFC 9309)
Opt out of AI training: one token for allShared policy token in robots.txt, support announced for 2027 (Google Search Central)Individual draft · 2027 plannedrobots.txtafter the editnew1User-agent: *2Disallow: /cart/34User-Agent: __usage_ai-training5Disallow: /6Allow: /press/78Sitemap: https://example.com/sitemap.xmlDeploySearch and indexing: unchangedThe group for all crawlers controls crawlingAI training: preference againstfor all paths except /press/applies once providers read the token!ChangelogWarningrobots.txt changed (+3 lines)+ User-Agent: __usage_ai-training+ Disallow: /+ Allow: /press/Pitfall: token line right below "User-agent: *"Groups merge, Disallow: / would then hit every crawlerSources: Google Search Central (Oct 30, 2026), IETF Datatracker (individual draft), RFC 9309. Illustrative example.
Anyone who wants to keep their content out of the training of generative AI models currently maintains a list in robots.txt: one group per crawler token, looked up again for every provider. Since October 30, 2026, there is a counterproposal: a shared signal for AI training that all participating providers are meant to read from one single group. Support was announced on the Google Search Central blog and is planned for 2027 (Google Search Central). It builds on the work of the AI Preferences working group (aipref) of the Internet Engineering Task Force (IETF), which is to standardize building blocks for preferences about AI use of content. Technically, the signal is an ordinary robots.txt group with the policy token __usage_ai-training in the User-Agent line, described in an individual draft without formal standing in the standards process (IETF Datatracker). This article shows what the token covers, what works in the meantime, how to add the group without hitting search crawlers, and how to keep an eye on the file after every deploy.

What was announced on October 30

The announcement is short and its key sentences are clear. Website owners are to be able to express their AI training preference for several AI providers in one step instead of maintaining separate instructions for every developer (Google Search Central). If a site uses the new token to opt out of training and does not explicitly name Google-Extended, the announcing provider plans to treat it as if Google-Extended had been disallowed. Google-Extended is that provider's own training token, offered since 2023 (Google Search Central).

Today, we're announcing our plans to support a shared AI training control in robots.txt.

— Google Search Central Blog, "Supporting a shared AI training control for websites", October 30, 2026

Two points matter for the assessment. First, the existing provider-specific tokens remain in place and keep working alongside the shared signal. Second, this is an intention with a timeline, not a fixed date: the provider aims to support the control together with other AI providers in 2027 and to report on progress (Google Search Central). The same provider's crawler documentation says early 2027 (Google Crawling Infrastructure).

Status: individual draft, not a standard

The token comes from the Internet-Draft draft-kelley-aipref-training-preference-signal-00. According to the Datatracker, it is an individual draft that is not endorsed by the IETF and has no formal standing in the standards process (IETF Datatracker). Internet-Drafts are valid for a maximum of six months and may be updated, replaced or withdrawn at any time; this one expires on May 3, 2027 (IETF Datatracker). Token name, rules and timeline may therefore still change before support arrives.

How the policy token works

The draft introduces no new directive and no new grammar. A site operator writes an ordinary robots.txt group whose User-Agent line does not name a crawler but a policy token for a purpose of use (IETF Datatracker). Path language, longest-match precedence and group structure stay as RFC 9309 defines them for the Robots Exclusion Protocol. Parsers that do not know the token skip the group like that of an unknown crawler. The example from the draft:

robots.txt (example from the draft)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The group is read like any other: where a path is disallowed, the operator has expressed a preference against AI training; where it is allowed, a preference in favor. In the example, that means no for the whole website and yes for the press material under /press-kit/. If there is no group with the token at all, no preference has been expressed (IETF Datatracker, Section 4.2). Four rules from the draft matter most in practice:

  • The group for all crawlers does not count. A group with User-agent: * must explicitly not be treated as a match for the token (IETF Datatracker, Section 4.2). A blanket Disallow: / for all crawlers therefore expresses no training preference, but it does block search.
  • Case does not matter, spelling does. The token is matched like any product token, case-insensitively but otherwise exactly (IETF Datatracker, Section 4.1). __usage_ai_training with an underscore instead of a hyphen is a different token and expresses nothing.
  • Not a crawling instruction. A provider may use the preference to avoid fetches, but does not have to (IETF Datatracker, Section 4.4).
  • Independent of crawler rules. Every crawler keeps reading its own group as before. Content usage and crawl control live in the same file but are separate from each other (IETF Datatracker, Section 4.3).

The draft does not define "AI training" itself. It takes the definition from the working group's vocabulary draft: using an asset to modify the learned parameters of a generative AI model (IETF aipref working group, vocabulary draft). Training models that perform exclusively non-generative tasks is not included. The vocabulary has a separate category for search; the token from the announcement does not touch it.

The new signal is narrow: it covers exactly one purpose of use. Whether a page is crawled, indexed and shown in search results is still governed by the groups for the crawlers themselves, usually the User-agent: * group. For Google-Extended, the token onto which the announcing provider maps the new signal, the documentation states explicitly that it does not affect inclusion in web search and is not a ranking signal (Google Crawling Infrastructure).

One caveat still belongs on the table before you set the opt-out. According to the documentation, the same token controls not only the training of future model generations but also so-called grounding: content from the search index is passed to a model at prompt time so that answers in the provider's AI applications are more factual (Google Crawling Infrastructure). The announcement notes that websites which explicitly allow the token are eligible to appear in such answers and receive clicks (Google Search Central). If you opt out via the shared token without naming that token explicitly, expect this route into the provider's AI answers to disappear as well. That is a trade-off between protecting content and AI visibility, not a technical question. How presence in AI answers can be measured in the first place is covered in the article on measuring Generative Engine Optimization (GEO).

Rule in robots.txtEffect todayEffect after announced support
User-agent: * with Disallow: /blocks crawling for all crawlers without their own group, including searchunchanged; explicitly does not count as a training preference
Group per training token, such as Google-Extendedworks with the provider that documents the tokenkeeps working and takes precedence over the token at the announcing provider
Group __usage_ai-training with Disallow: /skipped by current parsers as an unknown crawleropt-out of training at participating providers, treated like a Google-Extended opt-out at the announcing provider

What already works today: tokens per crawler

Until support actually arrives, only what providers document today has any effect: their own tokens per crawler or per purpose. Google-Extended has no separate user agent in requests and is read only in robots.txt as a control token (Google Crawling Infrastructure). Other providers separate training and search through different crawlers. OpenAI, for example, explains that disallowing GPTBot indicates the content should not be used for training generative AI foundation models, while a separate search crawler handles appearance in search results (OpenAI). A robots.txt that opts out of training with these two providers today while leaving search open looks like this:

robots.txt (current state)
# Search and indexing: group for all crawlers
User-agent: *
Disallow: /cart/

# AI training: one group per provider
User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

This list grows with every provider and is only as complete as your last look at the documentation. This is exactly where the shared token comes in: it is meant to close the gap for providers you do not list individually. Which tokens are common and how to recognize them in the access log is covered in the article on llms.txt and AI crawlers; how to measure their load is covered in the article on tracking AI crawler traffic. You do not have to type the per-crawler groups by hand: the free robots.txt and llms.txt generator creates them for the common AI crawlers. It does not know the new token yet, so you add that group by hand.

Do not delete existing groups

According to the announcement, the provider-specific tokens continue to work alongside the shared signal (Google Search Central). The draft says specific expressions should take precedence over general ones and allows providers to treat their own tokens as more specific than the shared token (IETF Datatracker, Section 4.5). Existing blocks therefore stay, and the token is added as a fallback.

Which stance fits which website

Whether opting out of training makes sense depends on the business model. The token supports paths, so it can also say no for individual sections. Three typical starting points:

Online store

Product and category pages should be mentioned in AI answers. Rather than a blanket opt-out, an opt-out for costly sections such as guides, via paths in the token group, makes more sense.

Specialist publisher and guides

The content is the product. An opt-out for the whole website with exceptions for press sections matches the example in the draft. Whether AI answers still link to the site belongs in the trade-off.

Agency with many clients

Every client website has its own robots.txt. Anyone adding the token for clients needs evidence per website of when the group went live, and a notification when it disappears.

Exceptions follow the familiar robots.txt logic: the rule with the longest match wins, and for equivalent rules Allow should take precedence (RFC 9309). An Allow: /press/ next to Disallow: / therefore works as expected. An Allow: /press without a trailing slash, by contrast, also covers a path such as /press-distribution-internal/, which is then released for training.

Adding the new group without hitting search crawlers

Because the token is just another group, its position in the file decides whether it works as intended. Under RFC 9309, a group consists of one or more User-Agent lines followed by rules; it only ends when another User-Agent line follows a rule (RFC 9309). This leads to the most common mistakes when adding it:

  • The token line directly below another User-Agent line. If User-Agent: __usage_ai-training sits immediately below User-agent: *, both form one group. A Disallow: / below then applies to every crawler without its own group, including search engines (RFC 9309).
  • A sitemap line as a supposed separator. Other records such as Sitemap: do not terminate a group (RFC 9309). Google shows exactly this case: two User-Agent lines with a sitemap line between them count as one group, and the following Disallow: / hits both crawlers (Google Crawling Infrastructure).
  • Blank lines do not separate. Under the RFC 9309 grammar, a blank line does not end a group either. Only a rule separates reliably: if the preceding group has at least one Allow or Disallow line, the token line starts a new group.
robots.txt (right: a group of its own)
User-agent: *
Disallow: /cart/

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

# Fallback for all other providers that read the token
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/

Sitemap: https://www.example.com/sitemap.xml
robots.txt (wrong: token slips into the group for all)
# Wrong: no rule between the User-Agent lines
User-agent: *
User-Agent: __usage_ai-training
Disallow: /

# Result: just one group, Disallow: / applies to every crawler

Two further limits: Under RFC 9309, crawlers must be able to parse at least 500 kibibytes (KiB) of a robots.txt (RFC 9309); Google parses up to 500 KiB and ignores whatever comes after (Google Crawling Infrastructure). In very large files, a group appended at the end may otherwise land beyond the limit. The file also has to be served as UTF-8 plain text; if the server returns an HTML page under /robots.txt, the parser tries to extract rules and ignores the rest (Google Crawling Infrastructure).

After adding the group, it pays to check the group structure before the file goes live. The free robots.txt and sitemap validator loads the file, splits it into User-Agent blocks with their rules and thus shows which lines belong together. General failure patterns, from an accidental Disallow: / to blocked resources, are covered in the article on common robots.txt mistakes.

When a change takes effect: cache, errors, redirects

A rule in robots.txt does not take effect the moment you deploy. Crawlers may cache the file; RFC 9309 says a cached version should not be used for more than 24 hours as long as the file is reachable (RFC 9309). Google generally caches the file for up to 24 hours, longer in case of timeouts or server errors (Google Crawling Infrastructure). A new rule may therefore take up to a day, and a mistake stays in force just as long.

More important is what a crawler sees when the file is missing or the server fails. Both typically happen during a deploy:

Response to /robots.txtConsequence under RFC 9309Consequence at GoogleConsequence for the training rule
Client error (4xx), such as 404 after a build without the filecrawlers may access any resourceslike a missing robots.txt, no crawl restrictions (except for 429)disappears without replacement, as does every per-crawler group
Server or network error (5xx)crawlers must assume complete disallowno crawling for the first 12 hours, then the last good version for up to 30 daysthe old version stays in force, search pauses
Redirectfollow at least five consecutive redirectsafter five redirects treated like a missing fileonly works if the target lies within the chain

The details come from RFC 9309, Sections 2.3.1.2 to 2.3.1.4 (RFC 9309), and from Google's interpretation (Google Crawling Infrastructure). For the training rule, the first case is the tricky one. A deploy that does not ship robots.txt removes not only the new token but every per-crawler block, and no crawler will tell you about it. The second case hits search: if /robots.txt returns a server error, Google initially stops crawling (Google Crawling Infrastructure). How easily this happens in a release process is shown by the related pattern of a noindex leaking in a deploy.

Monitoring robots.txt after every deploy

A rule that only takes effect in 2027 is written today and has to survive every deploy, every content management system update and every change of host until then. XICTRAQ® SEO monitoring fetches the robots.txt of every monitored website on a schedule and compares it with the most recently stored version. When the content changes, the changelog gets an entry with severity warning that lists the added and removed lines together with their count, such as +3 for three added lines. That shows from which fetch onward the token group was in the file and whether a later deploy removed it again.

Lines in the changelog

If the fetched robots.txt differs from the previous version, an entry appears with a date and usually with the added and removed lines. The schedulable SEO report includes changes to robots.txt.

Alert on a full block

If the group for all crawlers blocks the entire website, XICTRAQ® raises a critical alert. A separate token group does not trigger it, a merged one does.

Check for the token line

An API check on /robots.txt verifies status 200 and whether the token line is in the text. If it fails twice in a row, the alert goes out through your configured channels.

The full-block alert works per group: it evaluates only the User-agent: * group and fires when that group contains Disallow: / without Allow: /. It therefore catches the case where the token line slips into the group for all, and stays quiet as long as the training group stands apart. It is delivered through the channels you have set up in notifications.

If the file disappears entirely, there is no new version to compare, and the line comparison stays silent. That is what the check for the token line is for: two assertions that must both pass; like the draft, the second ignores case:

assertions-robots-txt.json
[
  { "type": "status_code", "operator": "eq", "value": 200 },
  { "type": "text_contains", "value": "User-Agent: __usage_ai-training" }
]

For online stores, the announcement comes at an awkward time. Before Black Week, many campaign pages and tests go live, and each of these deploys can overwrite robots.txt. Add the training group separately from the campaign release and check the changelog entry before the campaign code follows. How to evaluate the tests from the same week properly is covered in the article on A/B tests in Black Week.

What remains open until 2027

The announcement describes a direction, not a finished standard. Four points are explicitly open:

  • Name and rules of the token. The draft is in its first version and may be updated, replaced or withdrawn before support arrives (IETF Datatracker).
  • Provider participation. It only takes effect with providers that actually read the token. The announcement speaks of support together with other AI providers but does not name any of them (Google Search Central).
  • The rest of the vocabulary. The token uses only the definition of AI training; discussions on the remaining terms of the working group's vocabulary continue (Google Search Central).
  • Enforcement. Technical enforcement of preferences is explicitly outside the working group's charter (IETF aipref working group, charter). RFC 9309 also makes clear that robots.txt rules are not a form of access authorization (RFC 9309). The token is a statement, not a block.
Do not confuse: Content-Usage

The aipref working group has a second draft, draft-ietf-aipref-attach, which attaches preferences through a separate Content-Usage field in robots.txt and in HTTP responses (IETF aipref working group, attach draft). That is a different mechanism that the October 30 announcement does not mention. A Content-Usage line does not count for the announced signal.

Legally, the technical signal is only one building block. How a reservation of rights for text and data mining (TDM) is worded so that it holds up under German copyright law is covered in the XICTRON® article on the TDM opt-out for online stores.

Roadmap: prepare now, switch over in 2027

  1. Define the goal. No training, but search: for many site owners, that is the starting point. Also clarify whether you want to appear in AI answers, because depending on the provider that hinges on the same token as training.
  2. Take stock. Which per-crawler groups are in robots.txt today, and which of them concern training? The free AI readiness check shows which known AI crawlers are allowed or blocked.
  3. Add the token group. As a group of its own after a group with rules, not directly below another User-Agent line, with path exceptions only where they are intended.
  4. Check the structure. Have the validator split the file into groups and verify that User-agent: * has not gained a new block.
  5. Ship it separately. As its own small deploy, not bundled with campaign or template changes.
  6. Monitor. Keep the changelog and the full-block alert active, plus the check for the token line.
  7. Follow up in 2027. Check the Datatracker for a new version of the draft and follow provider announcements; if the token name changes, adjust the group.
One more group, not one list less

For the time being, the shared token replaces none of your existing rules. It is added as a fallback and takes effect in 2027 at the earliest, only with providers that read it. What it needs today is a robots.txt with cleanly separated groups and monitoring that makes changes visible.

Sources and status

This article is based on: Google Search Central Blog, "Supporting a shared AI training control for websites", October 30, 2026; Google Crawling Infrastructure, "List of Google's common crawlers" (Google-Extended section) and "How Google interprets the robots.txt specification"; IETF Datatracker, draft-kelley-aipref-training-preference-signal-00, individual draft dated October 30, 2026, valid until May 3, 2027; IETF aipref working group, vocabulary draft draft-ietf-aipref-vocab-08, attach draft draft-ietf-aipref-attach-05 and the working group charter; RFC 9309, Robots Exclusion Protocol; OpenAI, Overview of OpenAI Crawlers. Retrieved on November 2, 2026; product details as of November 2026.

What is the shared AI training signal in robots.txt?
A separate group in robots.txt with the policy token __usage_ai-training in the User-Agent line. A Disallow expresses a preference against AI training per path, an Allow a preference in favor. It is based on an individual draft without formal standing in the standards process (IETF Datatracker).
When does the token take effect?
Support has been announced for 2027, together with other AI providers (Google Search Central). Until then, current parsers typically skip the group like that of an unknown crawler. For now, only the tokens that each provider documents itself have any effect.
Will I lose rankings if I opt out of AI training?
For web search, the documentation does not foresee that: Google-Extended, the token onto which the announcing provider maps the shared token, affects neither inclusion in search nor ranking (Google Crawling Infrastructure). AI answers may be a different matter if a provider controls training and grounding through the same token. You should weigh that before opting out.
Should I delete my existing rules for individual AI crawlers?
Generally not. According to the announcement, the provider-specific tokens keep working (Google Search Central), and they are the only ones that work today. The token is added as a fallback for providers you do not list individually.
Can the token accidentally block search engines too?
Not the token itself, because the group for all crawlers explicitly does not count as a match (IETF Datatracker). What is risky is its position in the file: if the token line sits directly below User-agent: * or is separated from it only by a sitemap or blank line, the groups merge, and a Disallow: / typically hits every crawler without its own group (RFC 9309).
How do I notice that a deploy removed the rule?
By monitoring robots.txt. XICTRAQ® compares each fetched version with the previous one, records changes with the added and removed lines and alerts you when the group for all crawlers blocks the entire website. If the file disappears entirely, an API check that verifies the token line is in the response helps. No monitoring can promise to catch every case, but a silent loss typically does not go unnoticed for long.
Start in 60 seconds

Let XICTRAQ® watch your website.

Free plan with no fixed term available. No credit card, no installation, no hidden fees.