Cloudflare Allows Search Indexing, Blocks AI Model Training

Network security company Cloudflare has implemented new controls for AI-related bots, enabling website operators to differentiate between search engine indexing and AI model training. The updated system allows site owners to disallow the use of their content for training AI models, while still permitting legitimate search engines to crawl and index the site.

The new "Disallow AI Training" setting targets multi-purpose bots like Applebot, Googlebot, and Bingbot, which handle both search and potentially AI training tasks. Previously, blocking such a bot for training purposes would also have blocked its ability to index the site for search. Cloudflare's refined approach involves publishing these preferences in the robots.txt file and classifying bots by their intended use.

Purely AI training bots are already blocked by Cloudflare at its infrastructure level. For multi-purpose bots, the expectation is that they will respect the robots.txt directive against training while continuing to index for search. Cloudflare has categorized bots into three purposes: Search, Training, and Agent, providing more granular control.

These changes, effective September 15th, replace the previous blanket "Block AI Bots" option with more specific categories. Cloudflare intends to migrate existing configurations automatically. The company designates Apple, Google, and Microsoft as "accountable" entities, citing their provision of separate controls for AI training opt-outs or commitments to transparency and search visibility despite training restrictions.