One robots.txt Line, Four AI Giants, Zero Dollars: The AI Training Opt-Out Standardizes Refusal, Not Payment

The signal is one group in robots.txt: a User-Agent: __usage_ai-training line with Disallow: / for paths a site does not want used for foundational model training. It was defined in an individual IETF Internet-Draft posted to the AI Preferences working group list on October 30, 2026 by Kevin Kelley of Anthropic, with working-group chairs Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco. What the four companies agreed is a deployment promise, not the adoption of a standard — the draft itself carries no formal standing in the IETF process.

That line also defines what the token is not. It is a usage preference, not a crawl directive, and RFC 9309 crawling behaviour stays orthogonal to it, so a site that refuses training can still be crawled, indexed and quoted. The draft's most consequential sentence leaves every crawler that never joined the pact completely untouched, which is why a publisher still has to name-block each non-signatory crawler separately.

Nothing in the draft, in Google's announcement or in Microsoft's documentation creates a payment path, a licence or a floor price for training data. Every dollar of AI-content revenue still comes from bilateral deals, negotiated in the window where the labs need the corpus and the publishers need the traffic.

The two-sided strategy shows up in the filings. Reddit, which blocks crawlers that lack a licence, grew revenue 53.0% year over year to $895.0M in the third quarter of 2026; News Corp, which licenses archives while Dow Jones and the New York Post sue Perplexity, reported first-quarter fiscal 2027 revenue of $2.28B, up 6%. Publishers are monetising the refusal directly, not through the token — the token mostly makes that refusal legible to the four buyers who matter.

Every one of the four adopters is either a defendant in publisher litigation or a buyer of publisher content, and two of them control the search indexes that decide who sees a publisher at all. Alphabet and Microsoft are not neutral rulemakers here; they are the counterparties. The shared signal is drafted in a way that preserves the incumbent split between search visibility and training permission, and it carries a price tag: Google's announcement says a site that allows Google-Extended becomes eligible to appear in Gemini Apps responses and to receive clicks through grounding.

None of this makes the token useless. It makes it hand the labs legal cover while giving commercial leverage only to publishers who already have it — the opposite of what a neutral rulebook would deliver.

The token is voluntary; the network default is not. From September 15, 2026, Cloudflare ships a Disallow AI Training setting and starts new ad-monetising domains by refusing training crawlers on pages that carry ads. Mixed-use crawlers, the ones that both index for search and collect for training, are 36.6% of Cloudflare's verified crawler traffic, and just 17% of sites restrict training while fewer than 1% block search crawlers outright.

This is the layer investors should watch. Cloudflare's third-quarter revenue rose 39% year over year to $782.5M, it guided to roughly $2.96B for 2026, and on the third-quarter call its chief executive said agent payments could become the company's largest line. Upstream, Akamai's April 2026 research found AI bot traffic on its network rose 300% in 2025 and that training crawlers were 63% of the AI bots hitting media sites — identification, not consent, is the product being sold. A CDN default reaches every crawler that never joined the pact, which is how a voluntary standard becomes a mandatory one without a standards-body vote.

Robots.txt binds the crawler that fetches a page. It says nothing about who ends up owning the copy, or whether that crawler ever identifies itself. On November 2, 2026, The New York Times sued Brave Software in Manhattan federal court, alleging that Brave built a search index of more than 30 billion pages with a crawler it never disclosed, and sold access to that corpus to AI companies. Publishers cannot block a crawler they cannot name, and they cannot check it against the token either.

The consequence for the token is precise. Compliance value falls to the share of training corpora fetched directly rather than repackaged by an intermediary, and the cheapest way to defeat a standard is anonymity rather than disobedience. Anthropic, which already honours the token, is named in the same complaint as a buyer of the material, so how a lab proves the provenance of a corpus bought from a third party becomes a question a court will ask.

In Europe, the same line in robots.txt is about to carry legal weight no lab volunteers. On November 3, 2026 the Advocate General delivered his opinion in Case C-250/25, Like Company v Google Ireland, the first generative-AI reference to the Court of Justice. His view is that the text and data mining exception can cover AI training, but only if developers honour robots.txt opt-outs, stay away from pirate sources and show anti-memorisation measures — which turns a voluntary technical convention into evidence of lawful conduct.

The geopolitical wrinkle is worth naming. Cloudflare co-writes the syntax in an IETF working group; Luxembourg sets the legal weight. Europe is turning a convention engineered by four US companies into the test for lawful AI training on European soil, and publishers there will inherit whatever legal value the Court gives it.

Nothing about this changes next week's tape. The first observable effects are calendar events, not revenue events.