Google and Bing will back one shared robots.txt token for opting out of AI training, but the rule is only as strong as the crawler logs a publisher keeps to prove who obeys it.
Google and Microsoft’s Bing announced on 30 October that they will back one shared robots.txt token, taken from an IETF draft, that lets a site opt out of AI training. Google said it aims “to support this new control along with other AI providers in 2027” and will “provide updates on our progress.” PPC Land lists OpenAI as having published a commitment to the standard and reports that Anthropic is reportedly taking part in the standards work, which the IETF’s AIPREF working group is leading.
Bing is moving sooner and splits the choice three ways: AI training, AI grounding (using live pages to answer a question) and limited grounding. Temporary grounding controls appear in Bing Webmaster Tools on 15 December 2026, new preferences are honored from 1 February 2027, and the temporary settings expire on 30 September 2027.
The announcement turns a private preference into a shared rule, and a rule is only worth what a publisher can verify.
I learned this opening a whole city’s data: access works only when the rules of use are shared, public and enforceable. A common token meets the first two tests; the third still depends on evidence each publisher holds for itself. Expect the next round of licensing talks to treat compliance with the token as a contract term rather than a courtesy. The open question is whether anyone other than the publisher will enforce an opt-out that only the publisher can verify.
The UK Competition and Markets Authority has opened an investigation into whether Meta is abusing a dominant position by imposing “exploitative terms” on publishers and advertisers. Under the proposed terms, Meta could scrape and use content for AI training from any publisher website that has a business account. Meta has delayed the terms and pledged not to impose them until a full consultation is complete. Press Gazette reports the regulator is also looking at a trial that would make Facebook pages pay a Meta One fee to post external links, which is not yet a full investigation. No finding of wrongdoing has been made.
The Operator’s Take: Regulators now sit beside the Oslo court on one question: whether holding a business account can be treated as consent to AI use. A complaint is strongest when the publisher can show which pages Meta’s crawlers actually requested.
Legal · Industry Positioning Give your trade association dated records of Meta crawler requests to your site, to support its evidence to the regulator.
Digital Content Next filed traffic data in the New York federal case against OpenAI and Microsoft, where both sides have applied for summary judgment. The trade body says unique visitors to member sites fell 26% between 2023 and 2025, and that referred traffic fell 22% year on year in 2026, against 10% in 2025. Member advertising and total revenue each fell 5% in 2025, and advertising was 84% of revenue for the members surveyed. It wrote that retrieval is “already eating away at publishers’ businesses.” PPC Land notes the survey was published earlier and that its methods and sample sizes are not disclosed.
The Operator’s Take: Measured harm is what moves a court, but a members’ survey with undisclosed methods is not the same as a publisher’s own logs. The stronger exhibit is the one each publisher can produce for its own site.
Build · Detection & Telemetry Start keeping dated, site-level records of referrals from AI answers, so your own numbers can stand beside any trade-body survey.
NLA Media Access, which represents 14,000 UK titles, has launched a free advisory service that analyzes website data from participating publishers to find possible unpermitted scraping. It builds on the NLA Text Tracker, which it says has taken down more than 150,000 pieces of infringing content this year, including more than 40 full-site takedowns. Newsquest and three national publishers, including The Telegraph, are in the pilot. NLA already sells a license that lets organizations scrape publisher sites for a fee, and its chief executive expects “formal collective action” within 18 months if companies resist.
The Operator’s Take: The service pairs a license with the evidence to enforce it, which is the structure that makes a collective deal credible. A collective that can show who scraped what is negotiating from a count rather than a claim.
Commercial · Deal Execution Ask your licensing body whether its detection findings will be shared by crawler, so you can price access against observed use.
People Inc.’s chief innovation officer says the publisher is moving AI access from open crawling to storage buckets to custom APIs that deliver Markdown with author and update dates, because “90% of the content is junk meant for a browser to read, not an agent.” He added that “every token that needs to be processed is expensive.” The company is not charging for the API or selling ads to agents directly. Digiday reports it laid off about 6% of its workforce last week, and cites HUMAN Security data putting agent and agentic-browser traffic up 7,851% year on year.
The Operator’s Take: Years spent running hyperlocal news platforms inside a cable and broadcast company show that content keeps its value only where the operator owns the platform and the audience data around it. A custom feed is the first step toward pricing by use, but until it carries a price or a contract it is a free, better-formatted copy.
Commercial · Economic Model Decide what an agent request through a custom feed should cost, and attach that price to the terms of access before the feed opens to anyone.
The Times, which has about 680,000 digital subscribers behind a hard paywall, plans to test “macro payments” in the next couple of months, selling annual guides such as its rich list and best-places lists as one-off purchases. Its director of subscriptions said “more than 50% of web traffic is now bots and crawlers” and that there is no ready-made solution. Digiday reports that a Smartly performance tool cut cost per acquisition by more than 55%, and that over three quarters of occasional readers would find flexible access appealing.
The Operator’s Take: When bots are more than half of traffic, a page view no longer counts the reader a publisher is selling to. The Times is pricing the human part separately.
Commercial · Economic Model Separate human readership from machine traffic in your own reporting before you set any one-off or per-use price.