robots.txt: One Shared Signal to Opt Out of AI Training | Freewares

For twenty years, telling an AI company to leave your website alone meant one thing: adding its crawler to your robots.txt file, then doing it again for the next crawler, and the next. That game of whack-a-mole may finally be ending.

On October 30, OpenAI announced it will honor a shared signal in robots.txt that lets any website express, in a single rule, that it does not want its content used for AI training. Google said the same day that it intends to support the shared "policy token" too, with a target of 2027.

One rule instead of a dozen

Until now, opting out meant maintaining separate entries for GPTBot, Google-Extended, ClaudeBot, and whatever new crawler appeared next month. Miss one, and you were opted in by default. The shared signal comes from AIPREF, a working group at the IETF, the body that standardizes how the internet talks to itself. The underlying Internet-Draft is carried by Anthropic, with OpenAI committing to honor it and Google saying it intends to follow in 2027. Existing robots.txt directives keep working exactly as before; the shared token is an addition, not a replacement.

Why it matters

A single, standardized vocabulary shifts the burden. Instead of every webmaster tracking every crawler, the crawler makers agree to read one file, one way. It also turns the AI training opt-out from a courtesy extended by each company into something closer to a protocol, the kind of thing that is much harder to quietly walk back.

For small site owners especially, this is the difference between reading crawler documentation every few months and writing one line once.

What you can already do

The shared token is still being finalized, so the practical move today stays the same: declare your preferences per crawler. Blocking AI training currently looks like this:

User-agent: GPTBot
Disallow: /

Add Google-Extended and ClaudeBot on the same model. Once the AIPREF signal is final, one line should cover every participating crawler, and we will walk you through it when it lands.