A shared training preference begins to cross crawler boundaries
OpenAI and Anthropic support a common robots.txt signal for AI-training preferences, but their implementation timelines differ and the signal relies on crawler compliance.
OpenAI’s October 30 announcement backs a common robots.txt signal through which websites can express whether their content may be used for AI training.
The company says implementation will follow in the coming months, while existing crawler controls continue to apply. The announcement is a commitment to support the signal, not evidence that the change is already active everywhere. [1]
Anthropic’s documentation says it began recognizing the signal on October 30 and is rolling it out globally. Explicit rules for ClaudeBot take priority over a general training preference; controls for user-directed retrieval and search crawlers remain separate. A publisher should therefore inspect its existing crawler-specific rules before assuming a new general instruction changes the effective policy. [2]
The proposal is an IETF Internet-Draft, not a completed universal standard. It makes preferences easier to express consistently, but does not technically prevent noncompliant crawlers from fetching a page or remove material already used for training. Its value depends on adoption and correct interpretation by the services a publisher cares about. [1][2][3]
Sources & details 3 sources
Discuss in your voice app
Take the reporting, visual explanations and original sources into a conversation. Copy the brief, paste it into your preferred assistant, then start its voice mode.
Paste the brief into your voice app to discuss this reporting.Reporting sources
- OpenAI’s training-preference announcement
October 30 support announcement; implementation planned over coming months and existing controls retained.
- Anthropic crawler documentation
October 30 recognition and ongoing rollout; explicit ClaudeBot rules take precedence and other crawler purposes remain separate.
- IETF training-preference draft
Proposal status and semantics; Internet-Draft, not a final standard or access-control mechanism.
Images & figures
An explicit ClaudeBot rule takes priority over the general training preference; search and user-directed retrieval controls remain separate.
Euangelion · Source · Usage terms