OpenAI announced on Thursday that it will honor a shared signal in robots.txt, the file websites use to communicate how their content may be accessed, so that site operators can express a single AI training preference to multiple providers at once instead of maintaining separate rules for every crawler. Google and Microsoft are also supporting the effort, according to reporting by The Verge, with Search Engine Roundtable documenting adoption announcements from both companies on the same day.
The move addresses a long-standing pain point for publishers. Until now, a website that wanted to keep its content out of AI training pipelines had to identify each AI crawler individually and write separate rules for every provider, then keep that list current as new crawlers appeared. A shared signal collapses that maintenance burden into one line of intent, which participating companies agree to respect. For more context on this story, see our ongoing breaking AI news.
What OpenAI Committed To
In its announcement, OpenAI described the effort as voluntary and said it aims to implement the signal in the coming months, with implementation details to follow in a later update, according to Unite.AI's account of the announcement. The company said the new signal will not replace the controls it already provides through robots.txt, which will continue to apply and be respected.
OpenAI framed the work as drawing on the ongoing efforts of the Internet Engineering Task Force's AI Preferences working group, the standards body effort that has been working to define how websites can express machine-readable preferences about how their content is used in AI systems. The company said it is joining other working group participants in making the announcement and encouraged other AI providers to honor the signal as well.
Google and Microsoft Sign On
The significance of the announcement depends heavily on who else adopts the standard, and the early signs are that the largest players are in. Search Engine Roundtable reported on Thursday that Google has adopted the new AI training control, and separately covered Microsoft's Bing announcing new AI training controls connected to its AI search grounding products. The Verge reported that OpenAI, Google, and Microsoft are supporting the new standard for websites, which would make it the first cross-industry convention of its kind backed by the companies whose crawlers account for the bulk of AI-related fetching.
If the three companies follow through, a website operator would be able to set a single training opt-out that is understood by the crawlers behind ChatGPT, Google's AI products, and Microsoft's Copilot ecosystem simultaneously. That is a materially simpler world for publishers than the current one, where each provider documents its own crawler names and preference tokens.
Why Publishers Wanted This
The push for a unified signal has been building for years as AI training became a flashpoint between the technology industry and the people who produce the written record the models learn from. News organizations, in particular, have complained that AI crawlers scrape their reporting at scale while their advertising-funded business models erode, and several publishers have gone to court or signed licensing deals to reclaim leverage.
Interim solutions have proliferated. Cloudflare, which sits in front of a large fraction of the web, launched a setting that lets sites block AI training crawlers without losing Google search visibility, a distinction that matters because AI crawling and search indexing were historically governed by the same robots.txt rules. Standards work inside the IETF's AI Preferences working group has aimed to formalize preference expression at the protocol level.
A shared signal adopted by the major AI providers would be the most consequential step yet, because it works through the existing robots.txt mechanism that site operators already know how to use.
The Enforcement Question
The standard is voluntary, and that word carries weight. A robots.txt preference has no force of law by itself; it binds only the companies that agree to honor it, and compliance has historically depended on each provider's own detection and enforcement. OpenAI's statement that its existing robots.txt controls will continue to be respected suggests continuity rather than a new regime, and the company has not yet published the technical details of how compliance will be implemented or audited.
There are also open questions the announcement does not settle. It remains to be seen whether the signal covers only training, or extends to retrieval and grounding, the processes by which AI systems fetch live web content to answer questions. The distinction matters to publishers because retrieval-based products can drive traffic back to sources, while training use does not. How search-engine-managed AI experiences treat the signal versus their general crawling rules will shape whether publishers see adoption as a win or a fig leaf.
What to Watch
The next milestones are concrete: OpenAI's implementation update, the technical specification the participating companies publish, and whether other significant AI providers — including the large Chinese labs and independent inference companies — announce support. Publishers and rights holders will be watching most closely for the audit story, that is, whether operators can verify that their preference is actually being respected at scale.
If the standard holds, Thursday's announcement may be remembered as the moment AI training opt-outs stopped being a per-company patchwork and became a web-wide convention. If it stalls, it joins a long line of voluntary web standards that the industry applauded and then quietly ignored.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
