Major AI firms adopt unified opt-out standard for data scraping
OpenAI, Google, Microsoft, and Anthropic have committed to a shared standard allowing website publishers to block AI training crawlers via a single robots.txt signal.
Major technology firms including OpenAI, Google, Microsoft, and Anthropic have agreed to support a unified mechanism for website operators to restrict AI training data collection. The new approach simplifies a process that previously required individual configurations for every AI service provider.
The proposed solution involves a shared policy token integrated into a website’s robots.txt file, the standard protocol used to dictate crawler access. By applying this unified signal, publishers can now manage their content usage preferences across multiple platforms simultaneously rather than updating individual exclusion lists.
The technical standard was proposed by Anthropic’s Kevin Kelley through the Internet Engineering Task Force (IETF). The move addresses growing tension between content creators and AI developers regarding intellectual property and data rights. By standardizing the opt-out procedure, the companies aim to reduce the administrative burden on publishers seeking to protect their data from automated harvesting.