Cloudflare Bot Fight Mode and the AI Crawler Setting that Costs You Citations
Key takeaways
|
Cloudflare is where most of the bot blocking on the internet actually happens now. It's also where most of the accidental AI-visibility damage happens, because the defaults changed in 2025 and marketing teams rarely see the dashboard.
I put Cloudflare in front of Wix sites for one reason: Bytespider ignores robots.txt and Wix has no server rules. That's a narrow job. The trick is doing that narrow job without letting Cloudflare's broader defaults do a much wider one.
What is Cloudflare Bot Fight Mode?
Bot Fight Mode is Cloudflare's free-tier bot protection. It identifies requests that appear automated and issues a computational challenge intended to slow or stop them. It applies broadly, can't be tuned to specific crawlers, and doesn't distinguish between scrapers you dislike and integrations you rely on. Cloudflare's verified bot list, which includes major search engines, is exempt. |
Bot Fight Mode is designed for a site getting hammered by unknown automation, and for that it works. For a B2B marketing site it's usually overkill and occasionally harmful. It has been known to interfere with legitimate tools, API integrations, uptime monitors, and some crawlers that aren't on Cloudflare's verified list.
Super Bot Fight Mode, on paid plans, adds categories (definitely automated, likely automated, verified bots) with per-category actions. That's more useful, but still a category tool, not a crawler tool. If your problem is "Bytespider specifically," neither mode is the right answer. A WAF rule is.
I leave Bot Fight Mode off. Everything I need is in two other places.
Does Cloudflare block AI crawlers by default?
Yes, for domains added since July 1, 2025. Cloudflare announced that new zones default to blocking AI crawlers, and CEO Matthew Prince later reported 416 billion AI bot requests blocked across customers in the following five months. More than a million customers had activated AI-crawler blocking by late 2025. If your domain was onboarded after that date, assume the block is on. |
This is the setting that matters. The July 2025 change made Cloudflare, by a wide margin, the largest single force removing content from AI training and retrieval. It was framed as protecting publishers, and for a newspaper with a licensing strategy that framing makes sense.
For a B2B technology brand, it's the opposite of the goal. Those crawlers are how buyers find you, and customer acquisition cost goes up when a channel goes dark. And everything in the Engine Optimization Matrix below the SEO row depends on those crawlers reaching you.
The AI crawlers overview covers what each one does. The GPTBot and ClaudeBot posts cover why blocking them costs more than it protects. The irony is that Cloudflare's own robots.txt research found only 37% of the top 10,000 domains on its network had a robots.txt file at all, and GPTBot was disallowed in just 7.8% of those. Site owners weren't choosing to block AI crawlers. The default chose for them.
Which Cloudflare settings should you turn off for AI visibility?
Three. In Security, then Bots, set the AI bot blocking option to do not block. Leave Bot Fight Mode off. In AI Crawl Control, review any per-crawler blocks and allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, and Applebot-Extended. Then confirm no managed robots.txt is injecting AI Disallow lines. |
Step through them in order.
AI bot blocking. Security, then Bots. There's a setting specifically for AI crawlers with options along the lines of block on all pages, block only on hostnames with ads, or do not block. Set it to do not block. This is the July 2025 default and it's the one that costs citations.
Bot Fight Mode. Same page. Off.
AI Crawl Control. This is Cloudflare's renamed AI Audit dashboard, and it's genuinely useful. It shows request volume by crawler, by operator, and by purpose, and it has a robots.txt tab that flags crawlers requesting paths you've disallowed. Use it to see who's actually hitting you before writing rules. Check that nothing in it is set to block the allow list above.
Managed robots.txt. Cloudflare offers to inject AI-related Disallow directives into your robots.txt automatically. If it's on, your live robots.txt may contain rules you never wrote. Turn it off and confirm the file at yourdomain.com/robots.txt matches what you set in the Wix editor.
Run this audit quarterly, and any time someone on the IT or security side touches the Cloudflare account. Both times I've seen a client's citation rate collapse, it was a security team enabling a default without telling marketing.
How do you write a Cloudflare WAF rule that blocks only bad bots?
Go to Security, then WAF, then Custom rules, and create a rule with an expression matching the user agents you want to block, joined with or. Set the action to Block. Name the bots explicitly. Never use a wildcard, a category match, or Cloudflare's AI bot list as the rule's basis. |
This is the whole reason Cloudflare is in front of the site. The rule I run:
(http.user_agent contains "Bytespider") or
(http.user_agent contains "PetalBot") or
(http.user_agent contains "MJ12bot") or
(http.user_agent contains "DotBot") or
(http.user_agent contains "BLEXBot") or
(http.user_agent contains "DataForSeoBot") or
(http.user_agent contains "ImagesiftBot")
Action: Block.
Strictly, only Bytespider needs this. The rest honor robots.txt and would stop anyway. I include them because the WAF stops the request before it costs anything, and because robots.txt compliance can lag. The reasoning for each is in the bad bots hub and the individual posts for PetalBot, MJ12bot, DotBot, and BLEXBot and DataForSeoBot. Leave AhrefsBot and SemrushBot out if you use either tool.
One caution. contains is a substring match, so "DotBot" won't accidentally catch anything else common, but be careful with short strings. Don't write a rule matching "bot". You'd block Googlebot.
How do you put Cloudflare in front of a Wix site?
Add your domain to Cloudflare, then change nameservers at your registrar to Cloudflare's. In Cloudflare DNS, recreate the A records and CNAME target Wix shows in your Domains settings, with the proxy enabled. Set SSL mode to Full. Recreate every other DNS record you had, especially email. Wix doesn't officially support third-party proxies, so verify the site still loads and the domain still shows connected in Wix. |
Wix supports connecting domains by pointing, meaning your DNS lives elsewhere and points at Wix's IPs. Cloudflare can be that elsewhere, and its proxy sits in the path. The sequence:
Add the domain to a free Cloudflare account. Cloudflare imports your existing records. Review them.
In Wix, open Domains and note the exact A record IPs and CNAME target Wix requires for your site. Use those, not values from a blog post.
In Cloudflare DNS, confirm those records exist with the orange proxy cloud on.
SSL/TLS mode: Full. Wix terminates its own certificate. Flexible causes redirect loops; Full (strict) fails because Cloudflare can't validate Wix's cert chain.
Recreate every DNS record that isn't Wix. That typically means your email sending subdomain, any verification TXT records, and MX. Miss one and inbound email breaks.
Change nameservers at the registrar. Wait for propagation.
Immediately do the three-setting audit above. New zone means the AI crawler block is on.
Write the WAF rule.
Confirm the domain still shows as connected in Wix and the site loads on both apex and www.
If you're not seeing Bytespider or another robots.txt-ignoring crawler, don't do this. The Wix robots.txt setup is enough, and if you have server access the general blocking guide shows the Apache and nginx equivalents of the WAF rule. Cloudflare solves a specific problem and adds a layer where things can go wrong.
How do you verify Cloudflare isn't blocking good bots?
Check three places after setup and quarterly after that. Cloudflare's Security Events log should show blocks only for the user agents in your WAF rule. AI Crawl Control should show GPTBot, ClaudeBot, and PerplexityBot requests succeeding with 200 responses. And Google Search Console and Bing Webmaster Tools crawl stats should be unchanged from before the migration. |
The Security Events log is the source of truth. Filter by action Block and confirm every blocked user agent is one you named. If you see GPTBot, ClaudeBot, PerplexityBot, or Bingbot in there, a default is still on somewhere.
AI Crawl Control's per-crawler view is the second check. You want to see the good crawlers succeeding. Cloudflare's data put ClaudeBot at roughly 20% of AI crawler traffic in June 2026, so if you're seeing zero ClaudeBot requests, something's blocking it.
Then the standard checks in Google Search Console and Bing Webmaster Tools. Bing matters doubly here because Bing feeds ChatGPT search. If Bingbot crawl volume dropped after the migration, you've cut a GEO path.
Cloudflare is a good tool. It's also a tool whose defaults now point in the opposite direction from a B2B brand's AI visibility goals. Know the three settings, write one narrow rule, and check the logs. If you want a second set of eyes on the config before you flip nameservers, book 30 minutes.
Frequently asked questions
What is Cloudflare Bot Fight Mode?
Cloudflare's free-tier bot protection. It challenges requests that appear automated. It can't be tuned to specific crawlers and may interfere with legitimate tools. Verified bots like Googlebot are exempt.
Does Cloudflare block AI crawlers by default?
Yes, for domains added since July 1, 2025. Cloudflare reported 416 billion AI bot requests blocked in the five months after the change.
How do I turn off Cloudflare's AI crawler blocking?
Go to Security, then Bots, and set the AI bot blocking option to do not block. Also review AI Crawl Control for per-crawler blocks and disable any managed robots.txt injection.
Should I use Bot Fight Mode on a B2B marketing site?
Usually no. It's a blunt instrument for sites under active automated attack. For blocking specific bad crawlers, a WAF custom rule is more precise and less likely to break integrations.
How do I block a specific bot in Cloudflare?
Security, then WAF, then Custom rules. Write an expression like (http.user_agent contains "Bytespider") and set the action to Block. Join multiple bots with or.
Can I use Cloudflare with a Wix site?
Yes, by connecting your domain to Wix via pointing and proxying through Cloudflare. Wix doesn't officially support third-party proxies, so recreate all DNS records carefully and verify the site loads.
What SSL mode should I use for Cloudflare in front of Wix?
Full. Flexible causes redirect loops and Full (strict) fails certificate validation against Wix.
What is Cloudflare AI Crawl Control?
Cloudflare's dashboard, formerly AI Audit, showing AI crawler activity by crawler, operator, and purpose, including a robots.txt tab that flags crawlers ignoring your directives.
Which AI crawlers should I allow in Cloudflare?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, and DuckAssistBot, plus Googlebot and Bingbot.
How do I confirm Cloudflare isn't blocking Googlebot or GPTBot?
Filter Security Events by action Block and confirm only your named user agents appear. Check AI Crawl Control for successful GPTBot and ClaudeBot requests. Compare Google Search Console and Bing Webmaster Tools crawl stats to pre-migration levels.
