Does Bot Preference Sync overwrite an existing robots.txt?

No. Cloudflare states that if a site owner already has a robots.txt file, the content Bot Preference Sync adds is prepended to the existing material, so any Disallow directives already in the file are kept. The generated block sits between a BEGIN Cloudflare Bot Preference Sync marker and a matching END marker, leaving the rest of the file untouched.

How much does Bot Preference Sync cost?

Bot Preference Sync costs nothing extra. Cloudflare says the feature is available to all customers from the Free tier to Enterprise. It is on by default for new customers, and no crawler is blocked until the site owner picks a policy for the Search, Agent or Training categories in the dashboard.

Can I block AI training but still let search engines index the site?

Yes. Setting Training to Disallow writes a no-training preference into robots.txt. Cloudflare explains that cooperating mixed-use crawlers which take the extra Transparency step can still reach the content for search indexing, so a publisher can refuse training use without disappearing from search results.

What happens to sites already using Cloudflare's older managed robots.txt?

Existing users of the legacy managed robots.txt feature are not switched over silently. Cloudflare says it will prompt those customers to review and confirm their preferences in order to transition to Bot Preference Sync when it launches, so the current file is not rewritten until the owner approves the new settings.

Cloudflare · 2026-08-21 · major

Bot Preference Sync — Cloudflare writes your robots.txt to match your bot rules

Bot Preference Sync generates and updates your robots.txt from the AI bot policy you set in the Cloudflare dashboard, so the published file and the rules enforced at the edge stay the same. Free through Enterprise.

Cloudflare Bot Preference Sync announcement header

Cloudflare builds your robots.txt out of the AI bot policy you already set in its dashboard.

Quick facts

Maker	Cloudflare
Availability	All plans, Free to Enterprise
Bot categories	Search, Agent, Training
Search / Agent options	Allow, block on ad pages, or block everywhere
Training options	Allow or Disallow
robots.txt handling	Prepended between BEGIN/END markers
Bot list source	Cloudflare BotBase, refreshed periodically

What is it?

Bot Preference Sync writes your robots.txt for you, based on the AI bot policy you pick in the Cloudflare dashboard. Cloudflare sorts AI crawlers by the job they are doing — Search, Agent or Training — and turns your choice for each into user-agent rules in the file. It is generally available on every plan, from Free to Enterprise.

How does it work?

The generated rules are prepended to any robots.txt you already have, wrapped in BEGIN and END marker comments, so your own Disallow lines survive untouched. Bot names inside that block come from BotBase, Cloudflare's database of tracked crawlers, and are refreshed periodically as crawlers appear or change their user-agent. Search and Agent each take one of three settings; Training is a simple Allow or Disallow.

Why does it matter?

A hand-maintained robots.txt drifts away from what a site actually enforces at the edge, and crawler operators read the file rather than the firewall. Cloudflare's stated goal is that 'what you say to the world and what you enforce at the edge are kept in sync.' For publishers, the Training setting also puts a no-training preference in writing without costing them search traffic.

Who is it for?

publishers and site owners

Frequently asked questions

Does Bot Preference Sync overwrite an existing robots.txt?: No. Cloudflare states that if a site owner already has a robots.txt file, the content Bot Preference Sync adds is prepended to the existing material, so any Disallow directives already in the file are kept. The generated block sits between a BEGIN Cloudflare Bot Preference Sync marker and a matching END marker, leaving the rest of the file untouched.
How much does Bot Preference Sync cost?: Bot Preference Sync costs nothing extra. Cloudflare says the feature is available to all customers from the Free tier to Enterprise. It is on by default for new customers, and no crawler is blocked until the site owner picks a policy for the Search, Agent or Training categories in the dashboard.
Can I block AI training but still let search engines index the site?: Yes. Setting Training to Disallow writes a no-training preference into robots.txt. Cloudflare explains that cooperating mixed-use crawlers which take the extra Transparency step can still reach the content for search indexing, so a publisher can refuse training use without disappearing from search results.
What happens to sites already using Cloudflare's older managed robots.txt?: Existing users of the legacy managed robots.txt feature are not switched over silently. Cloudflare says it will prompt those customers to review and confirm their preferences in order to transition to Bot Preference Sync when it launches, so the current file is not rewritten until the owner approves the new settings.

Try it

Set your Search, Agent and Training preferences in Cloudflare's AI Crawl Control dashboard.