AI Labs Adopt Standard for Web Data Training Control
The IETF proposal aims to allow publishers to restrict AI training data usage while maintaining search visibility.
Updated on Oct. 30, 2026 in Artificial Intelligence
Live Poll
Do you trust that AI companies will give creators real control over how their data is used?
Microsoft, Google, OpenAI, and Anthropic have committed to adopting a new Internet Engineering Task Force (IETF) standard for AI training controls. The announced tool aims to provide web publishers with a clear mechanism to manage how their content is used for generative AI training and grounding.
Why it matters
This standardization addresses the friction between maintaining search engine visibility and controlling proprietary data usage. By aligning on an IETF protocol, these companies seek to resolve the current dilemma where publishers feel forced to choose between indexability and data protection.
The proposed IETF standard replaces bespoke, firm-specific opt-out mechanisms with a unified protocol for web crawlers. It maintains publisher eligibility for search engine discovery while establishing explicit preferences for generative model training usage.
The players
Microsoft
A multinational technology corporation known for its Windows operating system, Azure cloud infrastructure, and deep partnership with OpenAI.
A global leader in search engine technology, cloud computing, and AI research through its DeepMind and Gemini model divisions.
OpenAI
An AI research organization focused on developing large language models like GPT and operationalizing generative AI at scale.
Anthropic
An AI safety and research company that builds the Claude series of large language models focused on constitutional AI.
Internet Engineering Task Force
A global, open standards organization that develops and promotes voluntary internet standards.
The details
The tool functions as a centralized control interface for web administrators to signal their data usage preferences to automated scrapers. By adhering to an IETF standard, these firms move toward a common language for robot.txt-style directives, allowing publishers to programmatically restrict AI ingestion without sacrificing their visibility in search results like Bing. This architecture is currently in the proposal phase, aimed at simplifying the existing, complex landscape of crawler-based data management.
Timeline
October 30, 2026: Microsoft, Google, OpenAI, and Anthropic announced the agreement to adopt the new standard.
The Tech Race
This move represents a shift from proprietary crawler-blocking methods toward an industry-wide consensus on data access rights. It sets a new benchmark for how AI labs interact with the open web, moving beyond competitive silos to standardized protocols.
Web administrators and content creators can expect a simplified workflow for managing their data across major AI platforms once the standard is implemented. The update ensures that sites can continue to appear in search results while selectively opting out of training processes.
The takeaway
The adoption of an IETF standard signals a maturation in the relationship between AI labs and web publishers. Observers should track the release of technical implementation documentation for this standard to see how specifically these controls translate into site-level settings.
Further reading
Learn more about the evolving ecosystem of content management and machine learning in our Artificial Intelligence section.
Live Poll
Do you trust that AI companies will give creators real control over how their data is used?
