Cloudflare Blocking AI Crawlers Set to Allow: Causes and Fixes

Identify the blocking feature in Security Events

Select the client's domain in Cloudflare and open AI Crawl Control. Filter by crawler, hostname, the time of the failed fetch, and the article's URL path to review responses for that page. For https://example.com/blog/article, set Hostname to example.com and Path to /blog/article. The Metrics tab shows HTTP status codes, which indicate whether responses succeeded or returned errors. (Cloudflare AI traffic analytics)

An earlier WAF rule may block a request without the failure appearing in AI Crawl Control analytics. Open Events under Security > Analytics and look for requests in the same time window to the same article path.

The event details give you the information needed to choose a fix:

• Service: The security feature that acted on the request.
• Action: What it did, such as Block or Managed Challenge.
• Rule name and ID: The rule to inspect or modify.
• IP address and User-Agent: Information to compare with the crawler provider's official details.

The User-Agent identifies the browser or bot the requester claims to be. A Managed Challenge verifies the requester and may return a challenge page in place of the article. Include challenge events in your investigation: filtering only for Block can miss requests held up by this verification step. (Cloudflare challenge pages and Managed Challenge)

Security Events shows the actions taken by Cloudflare's security features. Read Service and Action together to determine whether the fix belongs in a WAF rule or a bot protection setting. (Cloudflare Security Events)

Changes and exceptions for the WAF rule responsible

If a block or challenge event identifies a custom rule in Service, find its name or ID in Security rules. Inspect which part of the rule matched the request, such as its source country, URL path, or User-Agent, to understand why it prevented the fetch.

For a country restriction, compare the crawler's source country with the countries permitted by the rule. If a crawler you intend to allow is being blocked, amend the rule to exempt that traffic. Changing AI Crawl Control to Allow leaves this separate rule in place.

For Perplexity, identify requests using both the bot name and official IP ranges. Perplexity recommends matching the User-Agent for PerplexityBot or Perplexity-User with its published IP ranges. Use the provider's current documentation when configuring the rule, since those ranges can change. (Perplexity WAF configuration and IP ranges)

If a Block or Managed Challenge event identifies Managed Rules in Service, a predefined Cloudflare check is preventing access. You can create an exception for the specific managed rule responsible, scoped to the intended crawler, article hostname, and path. Matching requests then bypass that check. (Cloudflare managed rule exceptions)

Skip is the action used to bypass selected security checks. Selecting All remaining custom rules also bypasses later custom rules. When one custom rule is responsible, amending that rule to exempt the intended traffic keeps the other rules' checks in place. (Cloudflare Skip options)

Rule order matters as well. Once Block or Managed Challenge executes, the request stops progressing through subsequent checks. An exception placed later cannot undo that action, so verify that the exemption runs before the blocking rule. (Cloudflare security execution order)

Different exception options for Bot Fight Mode and Super Bot Fight Mode

If Service names a bot protection feature in the event that prevented the fetch, inspect that feature's settings. Check the full name: Bot Fight Mode and Super Bot Fight Mode differ in their support for Skip.

Feature responsible	How to let the intended traffic through
Super Bot Fight Mode	Use a custom rule matching the intended traffic to skip this feature.
Bot Fight Mode	WAF Skip rules cannot bypass it. Consider disabling it or switching to bot protection that supports exceptions.

For Super Bot Fight Mode, create a custom rule matching the crawler's requests and select Super Bot Fight Mode as the feature to skip. This exempts matching traffic from its checks. (Cloudflare bot protection false positives)

Bot Fight Mode runs through a separate mechanism, so adding a WAF Skip rule leaves its behavior unchanged. Once you have identified it as the cause, review the site's other protections with its administrator and decide whether to disable Bot Fight Mode or switch to a feature such as Super Bot Fight Mode that supports exceptions. (Cloudflare Bot Fight Mode settings and limitations)

Verify successful responses and article content after the change

After changing the settings, check requests from the same crawler to the same article to see whether the response has changed. Allowed requests in AI Crawl Control counts requests that received successful HTTP responses with status codes 200–299. Allow in AI Crawl Control's Action column is a configuration value; read it separately from the count of successful requests. (Cloudflare response metrics and Allowed requests)

Keep the crawler, hostname, and path consistent and record when you changed the settings. Site-wide success counts alone cannot tell you whether the crawler reached the article you fixed.

Also verify that the response contains the article, rather than a login screen or another page. Work with the site administrator to check that the HTML served to the crawler includes the article's headings and body text.

Review the robots.txt permissions too. Our guide to robots.txt settings for AI crawlers explains search and training bots and how to configure their access.

Review the updated page with Please Cite Me!

Identifying the security feature responsible for a failed fetch tells you which WAF rule or bot protection setting to change. Verify successful crawler responses in Cloudflare's records, then check that the page serves the article content.

Please Cite Me! is a reporting tool for SEO agencies that checks whether a page is properly published and whether its content is complete. Its page audits help you review access and whether readers can understand and verify the information on the page.

Try Please Cite Me! to review page access and content gaps after updating the site's Cloudflare settings!

Analyze a page with Please Cite Me! →