Google, OpenAI, Microsoft and Anthropic Back One robots.txt Line to Opt Out of AI Training

The four largest Western artificial intelligence companies have agreed to honour a single line in a website’s robots.txt file as a refusal of AI training, ending — at least in principle — years in which every publisher had to say no to each company separately. OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 that they will recognise one shared policy token, __usage_ai-training, that lets a site owner opt its content out of generative AI training across all four companies at once.

The announcement, first reported by The Verge and confirmed in company blog posts the same day, is the most significant act of coordination the AI industry has yet produced on the question that has shadowed it since the first large language models appeared: who gets to decide what the open web is used to teach machines. It is also, for now, a promise rather than a finished system. The shared signal exists as an Internet-Draft — an early-stage working document, not a formal standard — and only one of the four companies has actually switched it on. The others have given themselves months to follow.

For publishers, the practical change is simple. Today, a news site that does not want its reporting used to train AI models must maintain a separate robots.txt entry for every provider: Google-Extended for Google, GPTBot for OpenAI, ClaudeBot for Anthropic, and more besides, with each entry needing to be kept current as crawlers change names. Under the new arrangement, one entry does the work of all of them. A site owner writes the policy token where a crawler name would normally go, followed by a disallow rule, and can carve out exceptions with allow lines:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example, taken from the draft specification itself, expresses a preference against AI training for everything on the site except the press kit, which the owner is happy for models to learn from. The same mechanism works at the level of individual paths, so a publisher could, for instance, block training on its archive while permitting it on freely distributed material.

How the shared signal works

The mechanism is deliberately unglamorous. It invents no new grammar and no new directive names. It reuses the robots.txt format that has governed automated access to websites since 1994, when Dutch developer Martijn Koster devised it after crawlers kept overloading servers, and which the Internet Engineering Task Force finally standardised as RFC 9309 in 2022. The trick is that the User-Agent line, instead of naming a crawler, names a purpose: training. Any participating company reading the file is expected to interpret that group as an instruction about its own training crawlers.

The specification, titled “Machine-Readable Training Preference Signal” and authored by Kevin Kelley, a technical lead for standards at Anthropic, was submitted to the IETF on 30 October 2026 as draft-kelley-aipref-training-preference-signal-00. It is being developed inside the IETF’s AI Preferences working group, known as AIPREF, which is chaired by Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco. The draft expires on 3 May 2027 unless it is updated, and the IETF’s own datatracker notes that an individual Internet-Draft “is not endorsed by the IETF and has no formal standing in the IETF standards process” — a caveat that matters, because the entire arrangement rests on the voluntary compliance of the companies involved.

Three points in the draft deserve attention from anyone who runs a website. First, the token governs use, not access. As the draft puts it, “this policy token is not a directive for crawling behavior” — a crawler may still visit and fetch a page whose owner has opted out of training; it is simply expected not to feed that page into a model’s weights. Second, the familiar wildcard rule, “User-Agent: *”, does not match the token, so a site that currently blocks every bot expresses no AI-training preference at all until it adds the new group by name. Third, private agreements can override the signal, so a publisher that has separately licensed its content to an AI company is not affected by what its robots.txt file says.

The four companies have adopted very different timetables. Anthropic, whose employee wrote the draft, has already begun implementing support, according to The Verge’s reporting, although the company has not published an announcement of its own. OpenAI said in its blog post that it aims to implement the signal “in the coming months” and will publish a further update when it does. Microsoft will begin honouring the new preferences as it crawls from 1 February 2027. Google has committed only to supporting the token during 2027, without a more precise date. Google’s John Mueller confirmed that if a site uses the shared token without separately mentioning Google-Extended, Google will read that as a Google-Extended block, which covers Gemini training while leaving the page in Google Search.

OpenAI, in its announcement, described the mechanism as a practical step toward an open web in which website owners have clearer ways to communicate their choices, and encouraged other AI providers to honour the signal. The company stressed that its existing crawler controls will continue to apply alongside the new shared token. Microsoft, in its own post, called the work the product of more than a year of effort by the IETF working group toward what it described as “a global harmonized practice on AI preferences”.

Microsoft goes further — and sets a deadline

Microsoft’s announcement, published on the Bing blog by principal product manager Krishna Madhavan, went considerably further than the shared token itself. The company is splitting the question publishers face into two separate, independent choices: training and grounding. Grounding is the step in which a model is pointed at live web content to inform an answer it generates — the retrieval behind AI overviews, Copilot responses and Bing’s AI Mode — rather than the absorption of content into a model’s weights during training.

Under Microsoft’s scheme, a publisher can set AI Training to Disallow in robots.txt while keeping AI Grounding set to Allow, so its pages can still inform generated answers, with links and citations pointing back to the original, without being used to train future models. Grounding preferences are set page by page, through a meta tag or an HTTP header, and Microsoft is introducing a middle option, DisallowExtended, which permits grounding only inside Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, while excluding the grounding services Microsoft sells to third parties, such as Web IQ and grounding through its Foundry API. There is no site-wide robots.txt setting for grounding.

Both preferences default to Allow when a publisher says nothing, and Microsoft is blunt about the trade-off: selecting DisallowExtended or Disallow for grounding “will significantly reduce” a site’s visibility in AI-generated experiences. At the same time, the company promises that neither choice affects eligibility for ordinary Bing search results. Neither choice, Microsoft adds, guarantees inclusion, traffic or payment.

There is also a deadline with real consequences. Microsoft’s NoArchive and NoCache tags, introduced in 2023 as the first publisher controls of their kind, will stop being read as training or grounding preferences after 1 March 2027. Publishers that currently rely on them must publish the new preferences before that date, and for grounding they must also record a temporary preference in Bing Webmaster Tools, which opens for that purpose on 15 December 2026. The temporary preference lasts until the page is recrawled or 30 September 2027, whichever comes first. If a publisher misses the March deadline, its pages default to Allow for grounding until Microsoft recrawls them and discovers a new setting — a gap that could leave content exposed for months.

What the opt-out does not do

For all its significance, the announcement leaves the biggest grievances of publishers untouched. The signal is voluntary. Robots.txt has always worked on trust: it is a polite notice on the door, not a lock, and nothing in the new draft changes that. A scraper that chooses to ignore the file can still take the content, and OpenAI itself frames the mechanism as “a coordination mechanism rather than a security control”. That distinction acquired fresh force this same week, when security researchers reported that a tampered copy of Anthropic’s Claude Code extension sat on Cursor’s code-editor marketplace for about eleven hours on 29 October, planting SSH backdoors and reaching up to 5,000 developers before it was pulled. Anthropic clarified that its own systems and official distribution channels were not compromised, but the episode was a reminder that the AI ecosystem’s weakest links are often not the big labs’ own crawlers.

The signal is also forward-looking only. Adding the token today does nothing about material already collected into existing datasets or already trained into model weights. Publishers who opt out now are blocking future training runs, not reclaiming what has already been taken — and the training data at the centre of the industry’s biggest legal fights, including the New York Times’ lawsuit against OpenAI and Microsoft, was gathered years ago under the old rules.

Nor does the shared token address the use of content that is costing publishers traffic right now. Opting out of training does not keep a page out of AI-written search answers, which quote and summarise pages without training on them. Reddit, for example, has seen more than a million weekly referrals from Google disappear as AI search drains its traffic, according to Reuters reporting this week — a trend that helped send its shares down 29 per cent on Friday. The token offers no remedy for that, except through Microsoft’s separate grounding control, and no equivalent exists yet at Google or OpenAI.

One major player is also missing. Meta, whose communications head Andy Stone welcomed the idea of a common signal, has not committed to honouring the token, according to The Verge’s report. Meta postponed enforcement of its own revised terms — which broaden the scope for using web content in its AI systems — from 30 October to 20 November, a delay that suggests the company is still deciding how far to go. Until Meta and the other large-scale data collectors, including Common Crawl, sign on, a publisher that adds the token has opted out of four companies, not of AI training as such.

The long road to this moment

The dispute the token tries to settle has been building since generative AI went mainstream. When ChatGPT arrived, publishers discovered that years of their work had been swept into training datasets with neither permission nor payment, and the first response was a patchwork. Google introduced Google-Extended in September 2023, letting sites block Gemini training while staying in search. OpenAI published its GPTBot crawler the same year. Cloudflare began offering one-click blocks on AI crawlers. Each company invented its own switch, its own name and its own syntax, and publishers that wanted to opt out everywhere had to maintain a growing list of entries in a file that was never designed for the job.

Regulators noticed. The European Union’s code of practice for general-purpose AI models, published under the AI Act, treats robots.txt as the principal machine-readable way for rights holders to reserve their content — a position Google cited in its own announcement. Courts are still working through whether training on copyrighted material without permission is fair use, with cases in the United States and Britain unresolved. The shared token does not settle any of those questions. What it does is remove the industry’s excuse that a common mechanism was technically impractical.

The timing is also revealing. The announcement came at the end of a week in which the AI industry’s economics were under unusual scrutiny: shares in the big technology companies slid sharply on Friday as investors questioned the returns on hundreds of billions of dollars of AI spending, and Anthropic filed registration documents for a public listing while securing a 63 billion dollar chip loan. A voluntary concession on training data costs the labs little in the short term — the token only governs future collection, and none of the four companies has to switch it on immediately — while buying goodwill with publishers, regulators and the courts at a moment when all three are watching closely.

Analysis: Why It Matters

1. A rare moment of coordination in a cut-throat race

The four companies backing this token are direct competitors spending hundreds of billions of dollars to out-build each other, and they have spent the past three years disagreeing in public about almost everything, from safety to regulation. That they have agreed on anything at all — let alone a mechanism that constrains how they collect their most important raw material — is itself the news. It suggests the labs have concluded that the fight over training data is a collective liability: every lawsuit, every publisher blockade and every regulatory intervention hurts all of them, and a shared opt-out is cheaper than a succession of separate settlements. The question is whether the coordination survives the first real test, when a publisher opts out and a rival model trained on that content anyway performs better.

2. The opt-out that is not quite an opt-out

Look closely and the mechanism’s limits are the story. It is voluntary, unenforced, forward-looking only, and — for three of the four companies — not yet switched on. A publisher that adds the token today gets a promise from Anthropic and a timetable from the rest. Nothing in the draft creates a way to check whether a company honoured the signal, to audit what was collected, or to punish a breach. That is not a criticism of the draft’s authors, who built the best mechanism the existing robots.txt format allows; it is a reminder that the format itself was designed in 1994 to stop crawlers overloading servers, not to enforce intellectual property rights in 2026. The token is a useful instrument, but it is not a settlement of the underlying dispute, and publishers that treat it as one will be disappointed.

3. Training is the wrong fight for many publishers

The most consequential choice in Microsoft’s announcement is the one it separated from training: grounding. For a news site, the commercial damage from AI is not mainly that a model trained on its reporting two years ago; it is that AI answers today summarise its reporting and keep the reader from ever visiting. Training is a slow, abstract harm. Grounding is an immediate one, measured in lost clicks and lost advertising revenue. By offering a separate grounding control — and by warning, honestly, that using it will reduce visibility in AI experiences — Microsoft has put the real trade-off in front of publishers for the first time: stay visible inside the answers, or keep your content out of them and accept the obscurity. Google and OpenAI have not yet offered the same choice, and until they do, the shared token answers the question publishers asked in 2023, not the one costing them money in 2026.

4. Microsoft’s two-track gamble

Microsoft has gone further than its rivals, and it has taken the bigger risk. By splitting training from grounding, retiring NoArchive and NoCache, and imposing a March 2027 deadline, it is asking publishers to make active choices — and to accept that choosing privacy means choosing reduced visibility. That candour is admirable, and it may also be strategic: a publisher that opts out of grounding in Bing’s ecosystem is opting out of the answers that increasingly define search, and Microsoft knows it. The company’s gamble is that transparency about the cost of opting out will keep most publishers opted in, while the existence of the choice defuses regulatory pressure. Watch whether Google follows with a grounding control of its own; if it does not, Microsoft’s approach will look less like an industry standard and more like a single company’s terms.

5. The standard is the beginning, not the end

An individual Internet-Draft expires in six months. For the token to matter beyond 2027, the AIPREF working group must turn it into a real standard, Meta and the other large collectors must adopt it, and someone must build the auditing and enforcement that robots.txt has never had. None of that is guaranteed. But the direction of travel is now visible: the era in which each AI company set its own rules for taking web content is ending, and the replacement — a shared, machine-readable statement of preference — is a framework that publishers, regulators and courts can all build on. The next test is whether the framework grows teeth, or whether it remains what it is today: four companies agreeing, politely, to read the same line.

What to Watch Next

1. Whether Meta commits to honouring the token before its revised terms take effect on 20 November, and whether Common Crawl and other large dataset builders follow.
2. OpenAI’s promised implementation update in the coming months, and whether Google gives a firm date for its 2027 support.
3. How many major publishers add the token to their robots.txt files in the next few weeks, and whether any large news group opts out publicly as a statement.
4. The IETF AIPREF working group’s progress toward turning the individual draft into a formal standard before it expires on 3 May 2027.
5. Whether Google introduces a grounding control to match Microsoft’s, and how publishers respond to the choice between visibility in AI answers and keeping content out of them.

Sources

• OpenAI — “Making it easier for websites to express AI training preferences” (30 October 2026)
• Microsoft Bing — “Your content, your choice: clearer preferences for training and grounding” (30 October 2026)
• IETF — “Machine-Readable Training Preference Signal”, draft-kelley-aipref-training-preference-signal-00 (30 October 2026)
• SiliconANGLE — “Google, Anthropic and others to support new AI crawling opt-out standard” (30 October 2026)
• Emergent — “Big AI Labs Back Shared robots.txt Token for Publisher” (30 October 2026)