OpenAI Now Honors Robots.txt for AI Training
Website operators can now selectively block AI training while allowing search indexing via standard site directives.
Updated on Oct. 30, 2026 in Artificial Intelligence
Live Poll
Do you believe websites should have the power to block AI companies from using their content?
OpenAI announced it will officially honor robots.txt signals to manage AI training data access. This policy enables website operators to independently configure crawling preferences for AI model training versus search functionality.
Why it matters
This change formalizes the mechanism for site owners to control how their content is utilized for AI training. It addresses a core tension between data accessibility and operator autonomy regarding automated scraping.
OpenAI distinguishes between two specific user agents: GPTBot for training and OAI-SearchBot for search. While operators can toggle access for these crawlers, the update does not retroactively remove data previously ingested into active training sets.
The players
OpenAI
An AI research and deployment company focused on building large language models and search tools.
The details
The robots.txt file functions as a standardized text file located at a website's root domain where operators define access directives. By configuring User-agent fields, site administrators can explicitly Disallow the GPTBot crawler while maintaining Allow status for OAI-SearchBot. These instructions act as server-side preferences to inform automated systems of preferred crawling behavior rather than functioning as hard-coded access authorization.
Timeline
October 30, 2026: OpenAI announced the new policy for robots.txt signaling.
The Tech Race
This policy aligns OpenAI's data collection methods with established web standards utilized by search engines for decades. It reflects a shift toward more granular control as AI labs face increasing pressure to provide transparency in their training ingestion pipelines.
Website administrators can now restrict training access for OpenAI models while keeping their content visible to search crawlers. These configurations take effect via standard server-side updates, though they apply only to future data ingestion.
The takeaway
This policy empowers site operators to distinguish between utility-based indexing and training-based ingestion for AI. Operators should monitor their logs to ensure that crawler behavior aligns with the updated robots.txt directives.
Further reading
Learn more about the infrastructure behind model development in our Artificial Intelligence section.
Live Poll
Do you believe websites should have the power to block AI companies from using their content?
