Four Labs Just Made Publisher Consent a Single Bit — and Left the Price to Someone Else

Kevin Kelley, Anthropic's technical lead for standards, wrote the specification four rivals adopted on October 30, 2026, filing it as an individual Internet-Draft with the IETF's AI Preferences work, intended for the Standards Track and expiring on May 3, 2027. The same day, OpenAI, Alphabet, Microsoft and Anthropic each published support for it, with Anthropic already in effect and OpenAI promising implementation in the coming months. Anthropic is also the outlier in how it buys content: rather than publisher licences it settled with authors, a reported $1.5 billion class settlement that received final court approval on July 20, 2026. The lab that settled over training rights now co-writes the file publishers use to withhold consent.

The draft contains no fee schedule, no metering rule and no payment extension. What it contains is a reversal: if a path is disallowed under the __usage_ai-training group, that expresses a preference against training use; if a path is allowed, it expresses a preference in favour. A wildcard User-agent: * block does not express a preference at all, so publishers have to write a dedicated group, and one group can carry several policy tokens for future purposes. Crucially, private agreements may override whatever the file says — the token is a preference, not a contract, and the contract is where the money already lives.

Those numbers explain what the bit is for. A publisher with one switch that four labs have promised to read can credibly withhold training from all of them at once, and a credible veto is worth money in a negotiation, because the alternative to signing is not no but not you, and the other three. The reported range runs from $50 million a year to $70 million a year per counterparty. The token makes the walk-away credible, and the walk-away is what the price is anchored to — which is why a standard four rivals could sign in a single afternoon leaves every fee schedule untouched.

Consent bits are worth what the network enforces. Cloudflare, which sits between crawlers and the sites they scrape, said AI training accounted for 52% of crawler requests on its network as of June 2026, up from 22% in spring 2025, and that mixed-use crawlers blending search, agent use and training exceeded 36% of that traffic. It now blocks AI training crawlers by default for new domains, and its pay-per-crawl product answers a crawler with HTTP 402 Payment Required and a price header, with the site owner setting the rate and the network acting as merchant of record.

Whoever reads robots.txt at the edge decides whether the consent bit means anything, and that party can also sell the exception. This is not one company's monopoly: Fastly announced in October 2026 that publishers on its network can classify AI bot traffic, publish machine-readable RSL licensing terms, and allow, block or charge individual crawlers. The edge layer is becoming a two-horse race in which the scaled operator captures the rent, and the labs have just made the race worth entering by agreeing that consent is a file.

The shared token governs training, and only training. Microsoft split the choice into two independent bits: AI Training, and AI Grounding, which decides whether content may inform generated answers. A publisher can therefore let its pages train a model and still keep them out of AI answers, and Microsoft's third option, DisallowExtended, keeps content inside Microsoft's own surfaces while excluding third-party grounding services. Alphabet chose the opposite asymmetry: opting out of the shared token is read as disallowing Google-Extended, but only sites that explicitly allow Google-Extended become eligible to appear in answers and take clicks via grounding in Gemini.

The pricing is where the thesis becomes testable. The edge trades at 39.9x sales and 265.0x trailing free cash flow, the labs at 6.1x to 11.5x, and the content owners at 2.0x to 5.2x with free cash flow yields of 5% to 7%. The market already pays a premium for enforcement and a discount for the assets enforcement governs, and that is precisely the pricing structure a re-priced training corpus should disturb. Alphabet's Q3 2026 filing shows why: Search and other revenue of $63.35B was 49.7% of $127.58B, while the Cloud segment took $30.71B at $11.12B of segment operating income. Training capacity and answer distribution sit in different segments of the same income statement, and the shared bit only touches the first.

None of this binds anyone. The standard is an individual Internet-Draft rather than a working-group document, and it expires on May 3, 2027 unless it advances toward the Standards Track. Meta signaled support for the shared signal but never wrote a robots.txt obligation into its commercial terms, which were pushed to November 20, 2026 pending a consultation with news publishers, even as a spokesperson told Nieman Lab that content owners can still use robots.txt to signal preferences. Only four providers announced support, and nothing in the announcement touches the crawlers that were not in the room.

Over one to three years, the valuable outcome is not that publishers block more but thatthe valuable outcome is a menu of deals rather than a wave of blocks. Once the same line means the same thing at four labs, an operator can put training on one switch and grounding on another, which is the precondition for selling access in layers: allow training and charge for grounding, allow both and collect the referral traffic, allow neither and license the archive outright. Microsoft already offers the two switches; Alphabet has not yet. The party that can execute that menu is the network, because the network sees every crawler request and a publisher only ever sees a file.

Fact: four providers committed to one token on October 30, 2026, one engineer's draft defines it, two of them have hard dates, and the largest network in the crawl path can already meter and price access. Fact: the token carries no price, a wildcard block does not express a preference, and private agreements override the file. Fact: AI training is 52% of crawler requests, up from 22% a year earlier. Inference: a credible cross-buyer veto raises the reservation price of every publisher licensing negotiation, because the alternative to a fee is a switch four buyers must honor at once. Speculation, and it is the weakest link in the chain: the rent ends up at the network rather than the newsroom, because the network can meter and charge every request while a publisher can only publish a file.