AI Training Control: New Shared Robots.txt Directives

The landscape of web content protection is shifting as major tech companies agree on a unified way to manage how your site is used for machine learning. Announced in late 2026, Google, Bing, OpenAI, and Anthropic are adopting a shared AI training control based on standards from the Internet Engineering Task Force (IETF). This article explains how this new "policy token" works, how it differs from existing directives, and the specific steps you need to take to implement it in your robots.txt file before the early 2027 rollout.

Understanding the New Shared AI Training Control

For years, website owners have faced a fragmented approach to managing AI crawlers. This often meant listing individual user-agents like GPTBot for OpenAI or Google-Extended for Google, and then updating these lists as new AI providers emerged. This method was time-consuming and prone to oversight.

The new solution aims to streamline this process with a single, shared signal. According to Google Search Central, this control is based on work from the AIPREF working group at the IETF. It allows websites to express their AI training preferences across multiple participating providers in a single step using a specific "policy token" within the robots.txt file.

Here is the core logic of this new standard:

• Universal Opt-Out: If you use the shared policy token to opt out of training, all participating providers (Google, Bing, OpenAI, Anthropic) plan to honor that request.
• Granular Override: Existing, company-specific controls, such as Google-Extended, remain active. According to Google, these specific controls will continue to function alongside the new joint signal. If you explicitly allow a specific provider while also using the shared opt-out, the specific instruction may take precedence. Google states that if a website uses the shared policy token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended.
• Training vs. Grounding: This is a crucial distinction. The shared control primarily targets training—the process of using your data to improve an AI model's underlying knowledge and weights. It does not automatically block grounding—the practice of using your live content to generate real-time answers or citations in AI experiences. You can block training while still allowing your content to be cited in AI-generated answers, depending on provider-specific grounding controls.

How to Implement the AI Training Control in Robots.txt

The implementation process is straightforward, requiring you to add a new block to your existing robots.txt file. According to updates from Google and Microsoft, the new control is identified by the user-agent __usage_ai-training.

You do not need to change your fundamental robots.txt syntax. The standard format of a User-agent line followed by Allow or Disallow directives still applies.

The Basic Syntax

To opt out of AI training across all participating providers, add a block like this to your robots.txt file:

1User-agent: __usage_ai-training
2Disallow: /

This directive instructs compliant crawlers not to use any content on your site for training AI models.

If you wish to allow training by default but restrict specific paths, you can modify the directive:

1User-agent: __usage_ai-training
2Allow: /
3Disallow: /private-portal/

This example allows training on all content except for the /private-portal/ directory.

Interaction with Existing Directives

Understanding how this new token interacts with your existing directives is essential for preventing unintended outcomes.

1. Google-Extended: Google has offered its Google-Extended user-agent for AI training control since 2023. According to Google, these controls function alongside the new joint signal. If you have Allow: / for Google-Extended and Disallow: / for the new shared token, Google will likely honor the specific Google-Extended allow for its own AI training. This means you can opt out of the industry-wide standard while still permitting Google to train on your content, or vice versa. To block Google's AI training, you must either remove any explicit Allow for Google-Extended or change it to a Disallow directive.
2. Bing (Microsoft) Controls: Microsoft is transitioning its AI content controls. While older directives like NoArchive and NoCache might temporarily influence grounding settings via Bing Webmaster Tools until September 30, 2027, they are being phased out for expressing AI training and permanent grounding preferences. Bing will begin honoring the new robots.txt preferences in early 2027, with a deadline of March 1, 2027, for publishers to configure new preferences if they wish to maintain restrictions previously set by older directives.
3. OpenAI and Anthropic: Both OpenAI and Anthropic have announced they will honor the shared robots.txt signal. This means you may no longer need to maintain separate entries for GPTBot or ClaudeBot solely for training purposes if you are using the shared token. However, it is prudent to retain specific Disallow directives for these bots if you intend to block other types of access (e.g., general crawling for search indexing by bots that might not yet support the new shared token).

Step-by-Step Implementation Checklist

Before modifying your live robots.txt file, follow these steps to ensure a smooth transition and prevent unintended blocking of legitimate crawlers.

1. Audit Current Directives: Download your current robots.txt file. Identify all existing AI-related user-agents you might be blocking or allowing (e.g., GPTBot, Google-Extended, CCBot).
2. Define Your Strategy: Decide your stance on AI training:
   • Total Opt-Out: Block all AI training for participating providers.
   • Selective Opt-In: Allow training for specific providers (e.g., Google) but block others via the shared token.
   • Training vs. Grounding: Determine if you want to block training but allow your content to be cited in AI answers (grounding). Remember, the shared __usage_ai-training token specifically targets training. For grounding, you may still need provider-specific controls or meta tags, depending on the provider's implementation.
3. Test in Staging: Create a test version of your robots.txt file in a staging environment. Do not upload it to production yet.
4. Add the Shared Token: Insert the User-agent: __usage_ai-training block with your desired Allow or Disallow rules into your robots.txt file.
5. Verify Syntax: Ensure there are no typos or misplaced characters. While robots.txt parsing is generally case-insensitive for user-agents, the user-agent string itself (__usage_ai-training) must be exact. Path matching is strict.
6. Consider Explicit Directives: If your strategy involves a mix of allowing and disallowing, or if you wish to allow training by default but restrict specific paths, ensure your directives are explicit. While robots.txt generally allows anything not explicitly disallowed, using clear Allow and Disallow rules, especially when combining the shared token with provider-specific ones, prevents ambiguity.

Timeline and Provider-Specific Details

Knowing when these changes take effect is critical for planning. The rollout is not instantaneous and varies by provider.

Bing Specifics: Microsoft has provided a detailed timeline for its new AI content controls. On December 15, 2026, temporary grounding controls became available in Bing Webmaster Tools for eligible sites using older NoArchive or NoCache directives. On February 1, 2027, Microsoft begins honoring the new robots.txt preferences for AI training and grounding. By March 1, 2027, publishers must configure new preferences to maintain restrictions if they were using old directives. The temporary grounding preferences expire on September 30, 2027.

Google Specifics: Google aims to support the new control along with other AI providers in 2027. They emphasize that this is part of a broader effort to foster an "open, creator-first web ecosystem." If you have previously opted out via Google-Extended, you do not necessarily need to change that unless you want to extend the opt-out to other providers via the shared token, or if you wish for the shared token to apply to Google's training as well.

Common Mistakes and Pitfalls

Even with a simplified standard, misconfiguration can lead to unintended outcomes. Here are the most common errors to avoid.

1. Confusing Training with Grounding

The shared __usage_ai-training token blocks training. It does not necessarily block grounding. If you block training, your content will not be used to improve the AI model's base knowledge. However, if a user asks a real-time question, the AI might still "ground" its answer by fetching your live page and citing it. If you want to prevent your content from appearing in AI answers entirely, you may need to use provider-specific grounding controls (like Bing's grounding settings or Google's grounding options) in addition to the training control.

2. Ignoring the "Explicit Specification" Rule

Google's documentation clarifies that if you use the shared token to opt out, and Google-Extended is not explicitly specified elsewhere in your robots.txt, Google will interpret this as a disallow for their AI training. However, if you have explicitly allowed Google-Extended (e.g., User-agent: Google-Extended followed by Allow: /), that specific instruction takes precedence for Google's AI training. This means you must review your existing Google-Extended rules and adjust them if you intend for the shared opt-out to apply to Google as well.

3. Forgetting to Update Bing Directives

Microsoft is phasing out reliance on older directives like NoArchive for AI purposes. If you are currently using NoArchive to block Bing AI, you must migrate to the new Bing Webmaster Tools settings or the new robots.txt preferences by March 1, 2027. Failing to do so may result in your restrictions expiring on September 30, 2027.

4. Syntax Errors in Robots.txt

A single misplaced space or typo in the user-agent string can render the rule ineffective. Always validate your robots.txt file using official testing tools (like Google’s Robots Testing Tool in Search Console or Bing’s equivalent) after making changes. Ensure the User-agent line exactly matches __usage_ai-training.

Verifying Your Implementation

How can you confirm that the new control is working as intended?

1. Crawler Logs: Monitor your server logs for requests from the new user-agent (__usage_ai-training). If you have set Disallow: /, you should observe a decrease or cessation of these requests after the implementation date (early 2027).
2. Search Console / Bing Webmaster Tools: Both platforms are expected to update their reporting to reflect the new signal. Look for new reports or indicators in the "AI" or "Crawling" sections that confirm the status of your AI training preferences.
3. Third-Party Tools: Many third-party tools and online validators can help you check your robots.txt syntax for errors, ensuring the new User-agent: __usage_ai-training block is correctly parsed.

Key Takeaways

• Unified Signal: The __usage_ai-training user-agent in robots.txt is the new industry standard for opting out of AI training, supported by Google, Bing, OpenAI, and Anthropic.
• Implementation Date: While announced in late 2026, full support and enforcement begin in early 2027. Bing starts honoring preferences on February 1, 2027.
• Granularity Remains: Company-specific controls (like Google-Extended) still work. You can mix and match, but explicit specific rules may override the shared signal for that provider.
• Training vs. Grounding: The shared token blocks training. It does not automatically block grounding (real-time citations). You need separate controls if you want to prevent your content from appearing in AI answers.
• Action Required: Audit your current robots.txt, decide your strategy for each provider, and prepare to add the new token in early 2027. Do not wait until the cutover date.

FAQ

Does this control stop AI from answering questions using my content? No. The shared __usage_ai-training control stops AI models from learning from your content (training). It does not stop AI from fetching your content to cite in real-time answers (grounding). To block grounding, you must use provider-specific settings or meta tags.

Can I allow training for Google but block it for everyone else? Yes. You can use the shared token to Disallow training for the general standard, while explicitly Allowing Google-Extended in your robots.txt. Google’s documentation indicates that explicit specific controls function alongside the new joint signal, allowing for such granular control.

What happens if I don’t update my robots.txt file? If you do not add the new token or adjust your existing directives, the default behavior will depend on each provider’s policy. Generally, if you have not explicitly disallowed access, AI providers will continue to crawl and potentially train on your content. For Bing, if you were using old directives like NoArchive for AI purposes, you must migrate to the new settings by March 1, 2027, to maintain your restrictions.

Is this control mandatory? No. It is voluntary. Publishers can choose to opt-in or opt-out. However, as more providers adopt this standard, it becomes the primary, most efficient mechanism for expressing preferences. Ignoring it means relying on older, provider-specific directives which may be deprecated or less comprehensive over time.