Google’s AI Training Opt-Out Token: Most Brands Should Skip It
Google, Microsoft, OpenAI and Anthropic will honour one robots.txt line that opts your site out of AI training. Here is who should use it, and why most Indian brands should not.
On 30 October 2026, Google, Microsoft, OpenAI and Anthropic each said they will honour one shared robots.txt line that asks AI companies not to train models on your pages. For most Indian brands our advice is blunt: skip the AI training opt-out for now, and make sure nobody adds it by accident.
It’s a training switch, not a search switch. It won’t pull you out of Google Search, and Google won’t support it until 2027. The bits that matter today are a quiet side effect on Gemini and a Bing deadline of 1 February 2027.
What does the AI training opt-out token actually do?
The AI training opt-out is a robots.txt group named __usage_ai-training. A Disallow rule under it asks every participating company not to use those pages to train generative AI models. It doesn’t block crawling for search, it carries no legal force, and a wildcard User-agent: * rule doesn’t count as a preference.
The token comes from an individual IETF draft filed on 30 October 2026 by Kevin Kelley of Anthropic. It borrows the AIPREF working group’s definition of AI training: using an asset to change a generative model’s learned parameters. No new directives, just the Allow and Disallow paths you already know.
The syntax, from the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That Allow line is the clever part. You can fence off your blog and still let models learn from your press kit or product specs.
Who has committed, and from when?
Four companies committed on 30 October 2026, each on its own clock. Microsoft starts honouring the token on 1 February 2027. Google’s crawler documentation says early 2027. Anthropic says it has already started implementing it, and OpenAI promises “the coming months”. Apple, Meta, Perplexity and Common Crawl haven’t committed to anything.
We compared the four statements side by side:
| Company | Starts | How it fits existing controls | The catch |
|---|---|---|---|
| Early 2027 | Read as a Google-Extended disallow if Google-Extended isn’t named | Also drops you from grounding in Gemini Apps and Vertex AI | |
| Microsoft Bing | 1 Feb 2027 | Adds page-level ai-training and ai-grounding meta tags | NoArchive and NoCache stop working as AI controls on 1 Mar 2027 |
| OpenAI | “Coming months” | Sits alongside its existing robots.txt crawler rules | Implementation details not published yet |
| Anthropic | Already implementing | Complements its own crawler controls | No firm date |
Meta is the obvious gap. Its revised AI crawling terms were pushed to 20 November 2026, as we noted in last week’s marketing roundup.
Bing dates to diarise: 15 December 2026 to 1 March 2027
Bing Webmaster Tools opens temporary grounding settings on 15 December 2026 for sites still relying on NoArchive or NoCache. Microsoft honours the new preferences from 1 February 2027. From 1 March 2027 those old tags no longer count as training or grounding restrictions.
Will blocking AI training hurt my Google rankings or AI Overviews?
Not your rankings. Google’s documentation says Google-Extended, the control this token maps to, has no effect on inclusion in Google Search and isn’t a ranking signal. AI Overviews and AI Mode run on Googlebot and aren’t covered by the token. The real cost lands in Gemini, because you also lose grounding in Gemini Apps.
Here’s the trap. Google-Extended governs two things at once: training future Gemini models, and grounding answers in Gemini Apps and Vertex AI. Google’s Search Central post, written by Search Advocate John Mueller, says a shared-token opt-out will be treated as a Google-Extended disallow unless you name Google-Extended yourself. So on Google you can’t say “no training, yes citations”. It’s both or neither.
Bing is more granular. Its grounding control has three values (Allow, DisallowExtended, Disallow), set per page. Krishna Madhavan, Principal Product Manager at Microsoft Bing, framed it this way: publishers “should not have to choose between being discoverable in Bing” and controlling AI use. Microsoft still warns that restricting grounding will cut your visibility in AI answers.
Training is yesterday’s argument. Grounding is tomorrow’s traffic. Don’t give up the second to protest the first.
Should an Indian brand add the AI training opt-out?
Most Indian D2C brands, SaaS firms, local businesses and agencies shouldn’t. Their pages exist to be found and quoted, and Google’s version also removes Gemini grounding. News, edtech, recipe, research and subscription publishers, whose text is the product they sell, have a real case to opt out of training while protecting search.
We use a three-question check we call the Train, Ground, Rank test:
- Train. Is your writing itself what customers pay for? If yes, the opt-out has value. If your content mainly sells a product, models learning your category language barely hurts you.
- Ground. Do leads or sales arrive when ChatGPT, Gemini or Copilot cite you? If yes, protect grounding. On Google that means leaving the token out until grounding is split from training. Our guide to getting cited by AI search covers the citation side.
- Rank. Nothing here moves Google rankings. Anyone selling the token as an SEO fix, or blaming it for a traffic drop, is wrong. With AI Overviews on 88% of commercial keywords, Googlebot access is the thing to guard.
Indian publishers have reason to care. ANI’s copyright suit against OpenAI, filed in the Delhi High Court in November 2024, put training rights on the Indian agenda. For a newsroom, one robots.txt line is a cheap, public record of what it wants.
Three robots.txt setups we’d ship
| Site type | What to put in robots.txt | Why |
|---|---|---|
| D2C store, SaaS, agency, local business | No __usage_ai-training group at all | You want AI assistants to learn and quote your brand |
| Publisher whose content is the product | User-Agent: __usage_ai-training, then Disallow: / | Blocks training across four providers; you accept losing Gemini grounding |
| Free plus paid content | User-Agent: __usage_ai-training, then Disallow: /premium/ | Protects what you charge for, keeps free pages working for you |
Check your live file: CDNs, SEO plugins and hosting dashboards can write AI rules into robots.txt for you. Open yoursite.com/robots.txt today. Existing GPTBot, ClaudeBot and Google-Extended rules keep their own meaning, and the new token never blocks crawling on its own.
To see which AI bots actually visit, Microsoft Clarity’s free AI visibility report shows bot and referral traffic. Decide with that data, not a hunch.
Where this plan can fail
It’s a polite request. The draft has no formal IETF standing and expires on 3 May 2027 unless revised, and its text only says a crawler “MAY” refrain from fetching disallowed pages. Nate Hake of Travel Lemming told Search Engine Roundtable the process isn’t done at the IETF level and significant disputes remain.
Coverage is patchy too. Apple, Meta, Perplexity and Common Crawl haven’t signed up, and Common Crawl’s archive feeds many models. TollBit found about 15% of AI page fetchers in Europe reached disallowed URLs in August 2026, according to PPC Land.
Our verdict also depends on Google. If it splits Gemini grounding from training before 2027, publishers could opt out of training at almost no cost, and we’d change our advice.
PingTalks verdict
Brands and agencies: leave the AI training opt-out alone, because AI citations are a growth channel and Google ties training to Gemini grounding. Subscription and news publishers: add it to paid or premium paths now, accept the Gemini trade-off, and revisit when Google ships in 2027. Everyone: audit NoArchive and NoCache tags before Bing’s 1 March 2027 cut-off.
What to do now
- Open your live robots.txt and remove any __usage_ai-training group you didn’t add on purpose.
- Run the Train, Ground, Rank test with whoever owns content and SEO, and write the decision down.
- If you use NoArchive or NoCache to limit AI, set a reminder for Bing Webmaster Tools on 15 December 2026.
- Publishers: put the token on /premium/ or paid paths first, not the whole site.
Frequently asked questions
Does the AI training opt-out remove my site from Google Search?
No. Google says Google-Extended, which the shared token maps to, does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode use Googlebot and are not covered.
Is User-agent: * Disallow the same as opting out of AI training?
No. Under the IETF draft, a wildcard group does not express a training preference. You need a group named __usage_ai-training for the shared signal to count.
When does Google start honouring __usage_ai-training?
Google’s crawler documentation says early 2027, and its Search Central post says 2027. Until then, Google-Extended remains Google’s control for Gemini training and grounding.
Can I block AI training but still be cited in Gemini?
Not on Google. Google-Extended covers both Gemini training and grounding in Gemini Apps, so blocking one blocks the other. Bing offers a separate page-level grounding control.
Do I still need GPTBot or ClaudeBot rules?
Yes, if you use them today. OpenAI and Anthropic both say the shared signal complements their existing crawler controls rather than replacing them.
Does every AI company honour the shared token?
No. Google, Microsoft, OpenAI and Anthropic have committed. Apple, Meta, Perplexity and Common Crawl have not, and the token is a voluntary request with no legal force.
Sources
- Supporting a shared AI training control for websites (Google Search Central Blog)
- Google’s common crawlers documentation
- Your content, your choice: clearer preferences for training and grounding (Microsoft Bing Blog)
- Machine-Readable Training Preference Signal (IETF draft)
- Google Adopts New AI Training Control (Search Engine Roundtable)
- Google Plans Support for Shared Robots.txt AI Training Control (Unite.AI)
- Google, Microsoft, OpenAI and Anthropic back one AI training opt-out (PPC Land)
Kousik BhattacharyaEditor, PingTalks
Digital Evangelist and Entrepreneur at Clickseek Digital. A results-oriented marketing leader with an 18+ year track record driving transformative growth for diverse global businesses. Expert in performance marketing, demand generation, customer acquisition, strategic partnerships, and market expansion within B2B SaaS, Consumer-Tech, Fintech, Travel/Hospitality, and CRM/ITES Solutions sectors. IIM alumnus with a strong analytical background and a string of accolades, including "Top 50 Most Influential E-Commerce Professional of India" and "Top 100 Digital Marketer of India."
