How to Block AI Crawlers With Cloudflare in 2026

Website owners no longer have to choose between leaving every AI crawler unrestricted and hiding useful pages behind a login. Cloudflare now provides a practical middle layer: observe AI crawler traffic, allow bots that create value, and enforce blocks against the rest.

This guide explains how to block AI crawlers with Cloudflare without accidentally removing your pages from traditional search. It reflects Cloudflare and Google documentation available on July 30, 2026, including Cloudflare's new Search, Agent, and Training policy categories.

The short answer

For most public websites, the safest starting policy is:

• Allow Search crawlers that send useful discovery or referrals.
• Block Training crawlers unless you have a licensing reason to allow them.
• Monitor Agent traffic first, then allow only the agents that support a real customer use case.
• Turn on managed robots.txt to state your preferences, but use AI Crawl Control or a WAF rule when you need enforcement.

That last distinction matters. A robots.txt file is a request that cooperative crawlers may follow. It is not an access-control system. Cloudflare's managed robots.txt documentation explicitly says compliance is voluntary and recommends AI Crawl Control for technical blocking.

What Cloudflare AI Crawl Control does

Cloudflare AI Crawl Control is available on all Cloudflare plans. It reports which identified AI services request your pages and lets you apply crawler-specific allow or block decisions. The product was formerly called AI Audit, so older tutorials may use that name.

Cloudflare separates AI bot behavior into three useful categories:

These categories are more useful than a single “block all AI” switch. A documentation site, for example, may want citations in AI answers but may not want its full archive collected for model training.

Cloudflare has also announced a change scheduled for September 15, 2026. For new domains, its updated defaults will block bots categorized as Training or Agent on pages that display ads, while Search remains allowed. Mixed-purpose crawlers will also be blocked by configurations that block AI training. Because this change was still upcoming on July 30, review your own zone settings instead of assuming the future default is already active.

Before you change anything: define the outcome

Blocking every crawler with “AI” in its description can reduce unwanted scraping, but it can also remove a discovery channel. Start with three questions:

1. Does this crawler send measurable referrals or citations?
2. Does it support a feature customers expect, such as an assistant retrieving current documentation?
3. Does its access create cost, competitive risk, or licensing concerns without giving anything back?

If AI referrals matter to your growth strategy, pair crawler controls with NextPJ's guide to Answer Engine Optimization so pages remain easy to cite for services you intentionally allow.

Step 1: confirm Cloudflare is actually proxying the domain

AI Crawl Control requires a Cloudflare account, a connected domain, and traffic proxied through Cloudflare. A DNS-only record bypasses Cloudflare's reverse proxy, so Cloudflare cannot enforce an edge block for traffic that never reaches its edge.

Open the Cloudflare dashboard, select the account and zone, and confirm that the relevant web hostnames are proxied. Also record a baseline before blocking anything:

• Normal search-engine crawl volume
• AI crawler requests and top requested paths
• Current referral traffic from search and AI services
• Origin bandwidth and cache-hit trends
• Any crawler that supports an existing commercial agreement

A baseline makes the change reversible. It also prevents a normal traffic swing from being mistaken for a policy result.

Step 2: review crawler activity in AI Crawl Control

Open AI Crawl Control for the zone and review the Security or Crawlers view. Cloudflare's current interface reports the crawler, operator, category, request count, unsuccessful requests, robots.txt violations, and selected action.

Filter by crawler name, operator, category, hostname, and path. Pay particular attention to:

• High-volume crawlers with no visible referral benefit
• Requests to archives, premium research, product data, or documentation
• Crawlers requesting paths already disallowed in robots.txt
• Unexpected spikes that increase uncached origin traffic
• Mixed-purpose crawlers whose training behavior conflicts with your policy

There is an important plan limitation. On the Free plan, identification relies on well-known, self-identifying user-agent strings, and the Metrics view is limited to the past 24 hours. Enterprise Bot Management adds stronger detection based on Cloudflare detection IDs and configurable timeframes. User-agent strings can be spoofed, so free-plan reporting should not be treated as perfect attribution.

Step 3: apply the narrowest effective policy

Cloudflare offers broad policy presets and per-crawler actions. Begin narrowly when the business impact is uncertain.

Recommended policy for a content or SaaS site

• Set Training to Block.
• Keep Search allowed if discovery and citations are valuable.
• Monitor Agent activity for one week, then allow known useful agents and block the rest.
• Add path-specific WAF exceptions when public marketing pages should be accessible but licensed archives should not.

To block one identified crawler, select it in AI Crawl Control and choose Block. According to Cloudflare's crawler management documentation, this creates or updates a WAF custom rule on the zone. That is enforcement at Cloudflare's edge, not merely a polite instruction.

Paid plans can customize the blocked response after at least one crawler is blocked. Cloudflare supports 403 Forbidden to state that access is denied or 402 Payment Required to indicate a licensing path. A plain, professional response with a licensing contact is more useful than an elaborate error page.

Do not advertise a paid crawling option unless you can actually process it. Cloudflare's Pay Per Crawl remained a closed beta as of July 28, 2026, even though the dashboard and documentation describe allow, charge, and block workflows.

Step 4: publish preferences with managed robots.txt

Enforcement and preference signaling solve different problems, so use both.

Cloudflare's managed robots.txt feature is available on all plans. If your origin already returns a robots.txt file with HTTP 200, Cloudflare prepends its managed section and preserves your existing rules. If no file exists, Cloudflare creates one. The managed section can disallow known training crawlers and include Content Signals.

Cloudflare's Content Signals Policy distinguishes:

• search for building a conventional search index and returning links or short excerpts
• ai-input for real-time model input, grounding, or retrieval in generated answers
• ai-train for training or fine-tuning models

Cloudflare is also testing a content-use extension with immediate, reference, and full reuse levels. Its managed default currently expresses search allowed, AI training disallowed, and reference-style reuse. These signals describe your intent; they do not stop a non-compliant scraper by themselves.

After enabling managed robots.txt, fetch the live file in a private browser window or with an HTTP checker. Confirm that:

• The response is HTTP 200.
• Your original sitemap and search rules remain present.
• Cloudflare's managed section appears only once.
• The intended AI training preference is visible.
• Important conventional search crawlers are not broadly disallowed.

Cloudflare notes that Google Search Console may report “Syntax not understood” for newer Content Signals directives, but says it has observed no effect on crawl rates or SEO from those warnings.

Avoid the Googlebot mistake

Do not block Googlebot merely because you want to restrict Google's AI use. Google's documentation identifies Google-Extended as a separate control token for use of crawled content in future Gemini model training and Gemini grounding. Google says restricting Google-Extended does not affect inclusion or ranking in Google Search.

That separation is exactly why category-aware rules are safer than a blanket bot ban. Keep conventional search access aligned with your SEO goals while controlling training and AI-answer use separately. Review Google's official crawler documentation whenever the tokens or affected products matter to a policy decision.

Step 5: verify that the block works

A dashboard toggle is not the end of the task. Verify results over the next 24 to 72 hours:

1. Check that blocked crawler requests receive the intended 403 or 402 response.
2. Confirm ordinary visitors and major search crawlers still receive normal pages.
3. Inspect robots.txt availability and Cloudflare's violations table.
4. Compare origin bandwidth, cache misses, and crawl volume against the baseline.
5. Check Search Console and server monitoring for accidental coverage changes.
6. Revisit referral analytics after one to two weeks.

Cloudflare calculates robots.txt violations by comparing current directives with past requests, not by logging each rule state in real time. A newly added disallow rule can therefore make older, previously legitimate requests appear as violations. Treat that table as a diagnostic signal, not unquestionable historical proof.

Optional hardening for persistent scrapers

If self-identified crawlers are only part of the problem, add controls gradually:

• Use a WAF custom rule for sensitive paths or additional user agents.
• Apply rate limits to abusive request patterns without penalizing ordinary visitors.
• Keep premium or personal data behind real authentication; crawler blocking is not data security.
• Consider Cloudflare AI Labyrinth for bots that ignore no-crawl guidance. Cloudflare says it inserts invisible, nofollow honeypot links that compliant bots ignore and that do not affect page appearance or SEO.
• Preserve logs long enough to distinguish a recurring scraper from a short-lived spike.

Do not depend on a user-agent string as proof of identity. Attackers can copy a legitimate crawler name. Where access matters financially or legally, use stronger bot detection, authentication, contractual controls, and professional legal advice as appropriate.

Limitations to understand

Cloudflare can control requests that pass through its network; it cannot retract content that was already collected. It also cannot guarantee that every scraper is identified, especially on user-agent-based detection. Robots.txt and Content Signals express preferences, but enforcement still depends on edge rules or access controls.

Blocking may also trade reach for control. An allowed AI search service might cite your article and send visitors, while a training crawler may deliver no direct referral. Measure these behaviors separately instead of treating all automated access as equivalent.

Finally, product labels and defaults are changing quickly. Cloudflare updated its crawler management documentation on July 28, 2026, and its planned September defaults were not yet active when this guide was written. Recheck the dashboard after major Cloudflare announcements.

Conclusion

The best Cloudflare AI crawler policy is selective, measurable, and reversible: allow search discovery that creates value, block unlicensed training by default, evaluate agents by use case, and combine robots.txt preferences with enforced edge rules.

Start with one zone, capture a baseline, block the clearest unwanted training crawlers, and verify both the HTTP response and SEO impact. That approach gives you meaningful control without accidentally making useful public content invisible.