When do Cloudflare's new AI crawler defaults take effect?

Cloudflare's new AI Traffic Controls defaults take effect on September 15, 2026 for ad-monetized pages. The granular Search, Agent, and Training switches are already available to every customer, so site owners can preview the change and opt in or out from the zone's Security Settings tab before the deadline.

Are training bots blocked by default on all sites?

The Cloudflare defaults block Training and Agent bots only on ad-supported pages after September 15. Search bots stay allowed by default so publishers keep their SEO traffic. Operators of any zone can flip the same defaults on manually today, whether or not the page carries ads.

What do AI bot operators need to change to keep crawling?

Cloudflare wants each crawler to declare its purpose using the Forwarded header (RFC 7239) with a use parameter — 'use=reference' for cited retrieval, 'use=immediate' for real-time agent actions. Cloudflare's managed robots.txt already emits use=reference by default; unlabeled bots on protected zones will match the Training category and be blocked.

Which AI companies does this affect?

Any AI vendor that crawls the web hits the new controls — OpenAI, Anthropic, and Google were named by Cloudflare in the announcement. Multi-purpose crawlers like Googlebot, Bingbot, and Applebot will be blocked on zones that disable Training, because today a single bot handles both search indexing and training data collection.

How does this connect to Pay Per Crawl?

Cloudflare paired the crawler controls with a Pay Per Use evolution of its earlier Pay Per Crawl marketplace, where publishers get paid based on how often their content actually appears in AI answers. Ceramic.ai and You.com are the first partners; the marketplace is separate from the free-tier bot controls and requires opting in.

Cloudflare · 2026-07-01 · major

Cloudflare — separate AI crawler controls for Search, Agent, and Training bots

Cloudflare adds three separate switches for Search, Agent, and Training AI bots on every zone. Site owners can allow search crawlers while blocking training and agent traffic. New defaults on ad-supported pages take effect September 15, 2026.

Cloudflare AI Traffic Controls dashboard splitting Search, Agent, and Training bots

Cloudflare replaces its single 'Block AI Bots' toggle with three per-purpose switches — Search, Agent, and Training.

Quick facts

Maker	Cloudflare
Announced	2026-07-01
New defaults take effect	2026-09-15
Bot categories	Search, Agent, Training
Default on ad pages	Search allowed; Agent + Training blocked
Applies to	New customers, new zones, and all free-tier users
Bot operator spec	Forwarded header (RFC 7239) with use= parameter

What is it?

Cloudflare's new AI Traffic Controls give site owners three independent switches for Search, Agent, and Training bots on every zone. The previous option was a blanket 'Block AI Bots' toggle that treated a Google search crawler and an OpenAI training scrape the same way. All Cloudflare customers, including the free tier, get the new controls immediately.

How does it work?

Each crawler is expected to declare its purpose in a Forwarded header (RFC 7239) with a use parameter — 'use=reference' means cited retrieval for an answer, 'use=immediate' means an agent is acting in real time on behalf of a user, and Training is any general data collection. Cloudflare's managed robots.txt now emits use=reference by default, and the BotBase dashboard shows Enterprise customers which categories are hitting their sites.

Why does it matter?

The AI Traffic Controls change how publishers and AI labs interact by default. Publishers can keep their content in Google Search while blocking OpenAI, Anthropic, and Google from training on it. Bot operators at those labs will have to tag every crawler by purpose or risk being blocked as unlabeled traffic once the new defaults land on September 15.

Who is it for?

publishers, AI labs running crawlers, and platform engineers who operate bots

Frequently asked questions

When do Cloudflare's new AI crawler defaults take effect?: Cloudflare's new AI Traffic Controls defaults take effect on September 15, 2026 for ad-monetized pages. The granular Search, Agent, and Training switches are already available to every customer, so site owners can preview the change and opt in or out from the zone's Security Settings tab before the deadline.
Are training bots blocked by default on all sites?: The Cloudflare defaults block Training and Agent bots only on ad-supported pages after September 15. Search bots stay allowed by default so publishers keep their SEO traffic. Operators of any zone can flip the same defaults on manually today, whether or not the page carries ads.
What do AI bot operators need to change to keep crawling?: Cloudflare wants each crawler to declare its purpose using the Forwarded header (RFC 7239) with a use parameter — 'use=reference' for cited retrieval, 'use=immediate' for real-time agent actions. Cloudflare's managed robots.txt already emits use=reference by default; unlabeled bots on protected zones will match the Training category and be blocked.
Which AI companies does this affect?: Any AI vendor that crawls the web hits the new controls — OpenAI, Anthropic, and Google were named by Cloudflare in the announcement. Multi-purpose crawlers like Googlebot, Bingbot, and Applebot will be blocked on zones that disable Training, because today a single bot handles both search indexing and training data collection.
How does this connect to Pay Per Crawl?: Cloudflare paired the crawler controls with a Pay Per Use evolution of its earlier Pay Per Crawl marketplace, where publishers get paid based on how often their content actually appears in AI answers. Ceramic.ai and You.com are the first partners; the marketplace is separate from the free-tier bot controls and requires opting in.

Try it

Open the zone's Security Settings tab in the Cloudflare dashboard to configure Search, Agent, and Training preferences.