Bad Bots vs Good Bots and Which Crawlers You Should Never Block
Key takeaways
|
Every week someone sends me a screenshot of their server logs or their Wix analytics and asks the same question. How do I stop all these bots?
After more than 16 years in B2B content marketing, my answer is usually the same. Don't. At least not all of them. About half the bots in that screenshot are the reason your brand shows up when a buyer asks ChatGPT for a vendor shortlist. The other half are scraping your site to sell backlink data to your competitors, or feeding a training pipeline that will never send you a click.
The skill isn't blocking bots. It's knowing which ones to keep.
What is a bad bot?
A bad bot is an automated crawler that consumes your server resources without returning any value to your business. It doesn't index you for a search engine people use, it doesn't retrieve your content to answer a real user's question, and it often ignores your robots.txt directives. Common examples include Bytespider, PetalBot, MJ12bot, DotBot, and BLEXBot. |
The word "bad" is doing a lot of work there, so let me be precise. I'm not talking about malicious bots in the security sense. Credential stuffers, DDoS botnets, and card testers are real, and Imperva counted them as 37% of all internet traffic in 2024. But they're a WAF problem, not a content marketing problem.
The bad bots I care about look legitimate. They announce a proper user agent. They read your sitemap politely. And they take your content somewhere that will never send a buyer back to you.
Three categories cover most of it.
SEO-data harvesters. MJ12bot (Majestic), DotBot (Moz), and BLEXBot and DataForSeoBot exist to build backlink indexes and SERP datasets sold by subscription. Your site is inventory. Whether that's bad depends on whether you or anyone you work with uses the tool. I keep AhrefsBot and SemrushBot because I use both platforms and want their data on my own site to be accurate, which matters when I'm running a keyword gap analysis or checking backlinks. I block the rest.
Search engines you'll never rank in. PetalBot crawls for Huawei's Petal Search, which has no meaningful footprint with B2B technology buyers in North America or Europe. SeekportBot, serpstatbot, and Barkrowler fall in the same bucket. The crawl cost is real. The upside is zero.
Training-data crawlers that don't retrieve. Bytespider is ByteDance's crawler and the clearest example. It's been widely reported to ignore robots.txt, it crawls aggressively, and there's no ByteDance product where being in the training data gets your brand named to a B2B buyer. That's the distinction from GPTBot or ClaudeBot, which I'll get to.
What is a good bot?
A good bot is a crawler that either indexes your content for a search engine people actually use, or retrieves your content in real time to answer a user's question. Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, and Perplexity-User are all good bots. Blocking any of them removes you from a distribution channel. |
Good bots map onto the four engines in the Engine Optimization Matrix.
SEO
Googlebot and Bingbot. Obvious, but Bingbot deserves more respect than it gets. Bing's index is what ChatGPT search retrieves from. If you've blocked Bingbot because "nobody uses Bing," you've also cut a retrieval path for ChatGPT. I covered this in the Bing Webmaster Tools guide and in why Bing is the underrated AI visibility lever, and it still surprises people.
AEO
The same two crawlers, because featured snippets and AI Overviews draw from the standard index. Google-Extended is the one to watch here. It's a robots.txt token, not a separate crawler, and it controls whether Google can use your content for Gemini grounding. Blocking it doesn't affect Search, but it does affect whether you're cited in AI Mode.
GEO
PerplexityBot builds Perplexity's index. Perplexity-User fetches a page when a real person's query triggers it. OAI-SearchBot does the same job for ChatGPT search. These crawlers determine whether your brand is named in a generative answer. They're also the ones most often caught by a "block AI bots" rule.
LLMO
GPTBot, ClaudeBot, and Applebot-Extended collect training data. Cloudflare's August 2025 analysis found training accounted for nearly 80% of all AI crawling, up from 72% a year earlier. This is where reasonable people disagree. My position for B2B brands is simple. You want the model to know your name, your framework, and your point of view.
Blocking training crawlers protects content with no resale value while removing you from the one place buyers increasingly ask for recommendations. The crawl-to-refer ratios look ugly (Cloudflare measured Anthropic at roughly 38,000 crawls per referral in July 2025, Perplexity at 194 to 1), but referrals aren't the goal for LLM visibility. Being the named answer is.
For the full breakdown of each AI crawler, start with the AI crawlers overview. This post is about the ones you should stop.
How do you tell good bots from bad bots in your logs?
Check three things for any unfamiliar user agent. Does the vendor publish crawler documentation with verifiable IP ranges? Does the crawler respect your robots.txt when you test it? And does the destination product ever surface your brand to a buyer? A bot that fails all three is a block. A bot that passes the first two but fails the third is a judgment call. |
Documentation
Every legitimate crawler operator publishes a page describing the user agent, its purpose, and how to verify it. OpenAI, Anthropic, Perplexity, Google, and Microsoft all do. Majestic and Moz do too. If you can't find a documentation page in two minutes, block it.
Robots.txt compliance
Add a Disallow rule, wait a week, check the logs. Polite crawlers stop. According to TollBit's Q4 2025 data, more than 13% of AI bot requests ignored robots.txt, a 400% increase from Q2. Bytespider is the headline offender, which tells you how much weight to give its stated purpose.
Return path
This is the one technical SEO guides skip because it's a marketing question, not an infrastructure question. Where does this crawler's data end up, and does a buyer ever see your brand there? For Googlebot the answer is a search result. For PerplexityBot it's a citation. For MJ12bot it's a row in a backlink database your competitor's agency pays for.
The server log analysis post walks through pulling user agents out of raw logs. If you're on Wix without log access, Google Search Console crawl stats and Bing Webmaster Tools give you the good-bot side, and a Cloudflare proxy gives you the rest.
Which bots should a B2B site block?
Block Bytespider, PetalBot, MJ12bot, DotBot, BLEXBot, DataForSeoBot, SeekportBot, serpstatbot, Barkrowler, and ImagesiftBot by default. Keep AhrefsBot and SemrushBot if you or your agency use those tools. Never block Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, or Applebot. |
This is the block list I run on mqlmagnet.com.
User agent | Operator | What it feeds | Verdict |
ByteDance | Model training | Block, needs edge enforcement | |
Huawei | Petal Search | Block | |
Majestic | Backlink index | Block unless you use Majestic | |
Moz | Link Explorer index | Block unless you use Moz | |
WebMeUp | SEO PowerSuite backlink data | Block | |
DataForSEO | SERP and backlink API datasets | Block | |
SeekportBot | Seekport | German search engine | Block |
serpstatbot | Serpstat | Backlink index | Block |
Barkrowler | Babbar | Link metrics | Block |
ImagesiftBot | The Hive | Image dataset | Block |
Ahrefs | Site Explorer index | Keep if you use Ahrefs | |
Semrush | Backlink and audit data | Keep if you use Semrush |
And the allow list. These never go in a Disallow block.
User agent | Operator | Engine | Why it stays |
Googlebot | SEO, AEO | Search index, snippets, AI Overviews | |
Bingbot | Microsoft | SEO, GEO | Bing index, ChatGPT search retrieval |
Google-Extended | AEO, LLMO | Gemini grounding, AI Mode | |
GPTBot | OpenAI | LLMO | Model training |
OAI-SearchBot | OpenAI | GEO | ChatGPT search index |
ChatGPT-User | OpenAI | GEO | Live fetch on user query |
ClaudeBot | Anthropic | LLMO | Model training |
Claude-SearchBot | Anthropic | GEO | Claude search index |
Claude-User | Anthropic | GEO | Live fetch on user query |
PerplexityBot | Perplexity | GEO | Perplexity index |
Perplexity-User | Perplexity | GEO | Live fetch on user query |
Applebot, Applebot-Extended | Apple | SEO, LLMO | Siri, Spotlight, Apple Intelligence |
DuckAssistBot | DuckDuckGo | GEO | DuckAssist answers |
Most sites get none of this right. Cloudflare found that only 37% of the top 10,000 domains on its network even had a robots.txt file in mid 2025, and GPTBot was disallowed in just 7.8% of those.
Which sounds like nobody's blocking AI crawlers. Until you remember Cloudflare made AI-crawler blocking the default for new domains on July 1, 2025 and reported 416 billion blocked AI requests in the five months after. The blocking is happening. It's just happening in a WAF setting marketing never sees.
How do you block bad bots without blocking AI crawlers?
Use robots.txt for the polite crawlers. Stack the user agents you want to block above a single Disallow rule, and leave your User-agent star group untouched so every unlisted crawler is still allowed. For crawlers that ignore robots.txt, proxy your site through Cloudflare and write a WAF rule that matches those specific user agents. Do not enable the blanket AI crawler block. |
Two layers, in order.
Layer one is robots.txt.
It's a request, not a wall, but most of the bad bots above honor it. The pattern is a named block below your default rules:
User-agent: Bytespider
User-agent: PetalBot
User-agent: MJ12bot
User-agent: DotBot
User-agent: BLEXBot
Disallow: /
Every User-agent line stacked above the Disallow inherits it. Your User-agent: * group stays separate and keeps allowing everything else. I wrote the robots.txt guide for AI crawlers for the allow side, and the Wix walkthrough covers where the editor lives and exactly what to paste. If you've also deployed llms.txt, keep the two files consistent. Nothing confuses a crawler faster than an llms.txt that invites it in and a robots.txt that turns it away.
Layer two is the network edge.
Bytespider won't read your robots.txt, so you need something that inspects the request before it reaches your host. On Wix, that means Cloudflare in front of the site. The Cloudflare guide covers the DNS setup and the one setting you have to turn off.
That setting is the trap. Cloudflare, and most bot management products, ship a one-click "block AI bots" option. It's the fastest way I know to remove a B2B brand from ChatGPT, Perplexity, and Claude simultaneously. Write your own rule. Name the bots. Never use the default.
Why does bot blocking matter for AI visibility?
Bad bots consume crawl budget and skew engagement data, but the bigger risk is overcorrection. Sites that block bots indiscriminately remove themselves from the retrieval paths that ChatGPT, Perplexity, Claude, and Google AI Overviews use to find and cite sources. Blocking is a distribution decision, not just a server hygiene decision. |
In the Engine Optimization Matrix, crawler access sits under the Distribution lever for GEO and LLMO. Distribution for those engines isn't a promotion channel. It's presence on the surfaces generative engines pull from, and that starts with whether their crawlers can reach you. Every crawler in the allow list is a discovery channel, and in B2B, discovery is the top of the customer acquisition funnel. Blocking one is a CAC decision made by someone who doesn't own CAC.
I've watched this go wrong twice in the last year with clients. Both times a security team enabled a bot management default without telling marketing. Both times citation rate dropped within weeks and nobody connected the two until someone checked the WAF logs.
The thesis I keep coming back to is that AI citations don't correlate with top-10 Google rankings. Pages get cited from position 40. Pages get cited that rank but never get cited for the wrong reasons too. None of it happens if the crawler was turned away at the door.
So treat bot blocking like any other distribution decision. Name the bot, know where its data goes, decide. The junk goes. The good ones stay, and you check crawl budget and crawl reports quarterly to make sure a well-meaning toggle didn't quietly undo the work.
If you want a second set of eyes on your robots.txt and WAF rules before you touch them, book 30 minutes and we'll go through your logs together.
Frequently asked questions
What is the difference between a good bot and a bad bot?
A good bot indexes your content for a search engine people use or retrieves it to answer a real user's question. Googlebot, Bingbot, GPTBot, and PerplexityBot are good bots. A bad bot consumes your server resources without returning traffic, citations, or visibility. SEO-data harvesters like MJ12bot and DotBot, and aggressive training crawlers like Bytespider, are bad bots for most B2B sites.
What percentage of web traffic is bots?
Imperva's 2025 Bad Bot Report found automated traffic reached 51% of all web traffic in 2024, the first time bots surpassed humans in a decade, with bad bots alone at 37%. Cloudflare Radar data from June 2026 put bots at 57.5% of HTML requests across its network.
Should I block AI crawlers like GPTBot and ClaudeBot?
For B2B technology brands, no. GPTBot and ClaudeBot collect training data that determines whether ChatGPT and Claude know your brand, your framework, and your point of view. Blocking them protects content with no resale value while removing you from the place buyers increasingly ask for vendor recommendations.
What is Bytespider and why can't robots.txt stop it?
Bytespider is ByteDance's web crawler. It has been widely reported to ignore robots.txt directives and to crawl aggressively. Because it doesn't honor Disallow rules, the only reliable way to block it is at the network edge, using a Cloudflare WAF rule or equivalent that matches its user agent before the request reaches your host.
Does blocking bad bots improve SEO?
Indirectly. Blocking bad bots reduces wasted crawl budget and server load, which can help Googlebot and Bingbot crawl more of your site more often. It also cleans up analytics. It doesn't directly raise rankings, and blocking the wrong bots will hurt you far more than blocking junk will help.
How do I know which bots are visiting my website?
Server logs show every user agent that requests a page. If you don't have log access, Google Search Console crawl stats and Bing Webmaster Tools show the good-bot side, and a Cloudflare proxy shows everything else in its analytics dashboard. Look for user agents you don't recognize and check whether the operator publishes crawler documentation.
Is bot traffic bad for my website analytics?
Most analytics platforms, including GA4, filter known bots automatically. The bots that slip through are usually headless browsers and referrer spam, which inflate sessions and distort engagement metrics. Server-side crawlers like MJ12bot don't execute JavaScript, so they rarely appear in GA4 at all.
Should I block AhrefsBot and SemrushBot?
Only if nobody on your team or at your agency uses those tools. Both crawlers honor robots.txt and both feed platforms many B2B marketers rely on for their own site data. Blocking them makes your own Ahrefs or Semrush reports less accurate.
What is Google-Extended and should I block it?
Google-Extended is a robots.txt token that controls whether Google can use your content for Gemini training and grounding. It doesn't affect Search rankings. Blocking it reduces your chance of being cited in Gemini and Google AI Mode responses, so B2B brands pursuing AI visibility should leave it allowed.
Does Cloudflare block AI crawlers by default?
Yes, for new domains added since July 1, 2025. That default blocks GPTBot, ClaudeBot, PerplexityBot, and other crawlers that feed AI citations. Check the AI Crawl Control setting and turn it off, then write a specific WAF rule for the bots you actually want to block.
Can I block bots on Wix?
Yes. Wix has a robots.txt editor under SEO and GEO in the dashboard, which handles polite crawlers. For crawlers that ignore robots.txt, you can proxy the site through Cloudflare and use WAF rules. Wix App Market blockers run client-side JavaScript and don't stop server-side crawlers.
How often should I review my bot block list?
Quarterly. New crawlers appear constantly, operators rename user agents, and security teams change WAF settings without telling marketing. A quarterly check of crawl stats in Google Search Console and Bing Webmaster Tools, plus a scan of your Cloudflare firewall events, catches most problems before they affect citation share.
