__usage_ai-training robots.txt: how to block AI training
2026-10-30 · 4 min read
__usage_ai-training is a new robots.txt user-agent that tells AI companies whether your pages can be used to train their models. To opt out, add a group with User-Agent: __usage_ai-training and Disallow: / to your robots.txt. Google, Microsoft, OpenAI and Anthropic have all said they will honor it. Anthropic is reportedly starting now, and Google and Microsoft say 2027.
Vanessa Fox walked through it on X on Friday. @vanessafox wrote that all four companies "are all working towards support (early 2027) of a new user-agent in robots.txt," and in a follow-up linked each company's announcement and said she had updated her checker tool to look for the new line.
What was announced
The token comes from an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal, filed October 30 by Kevin Kelley of Anthropic. People from Google, OpenAI and Microsoft are named in its acknowledgments, and each company posted its own plan:
| Company | What it said | When |
|---|---|---|
| Will support the shared signal; an opt-out with no Google-Extended rule is treated as a Google-Extended block | 2027 (Search Central post) | |
| Microsoft | Will honor it as Bing crawls, plus new page-level training and grounding tags | February 1, 2027 (Bing blog) |
| OpenAI | Published its own announcement; coverage reports support "in the coming months" | Not dated |
| Anthropic | Wrote the draft; reported to have started implementing | Already underway, per reports |
Until then, the per-company controls you may already use (GPTBot, Google-Extended and the rest) keep working. Leave them in place.
How to set it up
This is the example from the draft itself:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That says no to training on the whole site except /press-kit/. If you only want to protect part of the site, list just those paths:
User-Agent: __usage_ai-training
Disallow: /members/
Disallow: /courses/
Leaving the token out is read as allowing training. A few things to know before you edit the file:
- You have to name the token. The draft says
User-Agent: *"MUST NOT be interpreted as matching the policy token," so an existing catch-all group doesn't cover it. - It won't stop crawling. The draft describes a preference about how content is used, and bots can still fetch your pages. Rules for named bots like GPTBot still decide what gets crawled.
- Regular search results stay the same. Microsoft says these preferences don't change eligibility for normal Bing results, and Google-Extended doesn't affect Google Search rankings.
- Gemini citations can get caught up in it. If you opt out with the shared token and have no Google-Extended rule, Google will treat that as blocking Google-Extended, and that also controls whether Gemini Apps can cite and link you. If you want to stay out of training but still get those citations, add an explicit
Allowfor Google-Extended. - Bing adds page-level tags. A single page can override robots.txt with
<meta name="Bingbot" content="ai-training:disallow">. A separateai-grounding:disallowtag controls whether Copilot and Bing's AI answers can use the page. After March 1, 2027, Bing will no longer treatNoArchiveorNoCacheas a training or grounding preference.
It isn't finished
Nate Hake of Travel Lemming, who is thanked in the draft's acknowledgments, pushed back on the idea that this is settled. @natejhake wrote that "it's not actually done at the ietf level," and that "some significant areas of dispute remain." In another post he says he's "still quite unhappy about one thing in the current draft." His complaint is that the preferences basically don't apply to anything a user does, which he says hurts artists and creators getting copied by AI spam. He also pointed out that the IETF is open to anyone, and nobody needs an invitation to take part.
He's right on the status. It's an individual draft that expires in May 2027 unless someone revises it. My read: with Google, Microsoft, OpenAI and Anthropic all publicly committed to reading the same line, I don't expect the basic syntax to change much.
What to actually do
Most small businesses want AI tools to read their pages, because that's how ChatGPT, Gemini and Copilot recommend a plumber or a salon. Training is a separate question from being cited, and this line only covers training. A shop with nothing proprietary on its site can leave it alone. A business that sells courses, publishes original research or runs a members area should add the token now for those paths, so it's in place when the crawlers start reading it.
Training data has been in the news here before, in our post on OpenAI's offline training shift. And if an AI tool built your site, as in our look at ChatGPT's website builder, check that its robots.txt says what you actually want.
If you're not sure what your site tells crawlers now, or which bots you should let through, New Face Design's free process audit includes a robots.txt and AI-visibility check. We'll tell you plainly whether to change anything.