How to Block Bad Bots from Your Website: Tips to Help You Facilitate AI Crawlers
Key takeaways
|
Most "how to block bots" guides are written by hosting companies or security vendors, and they treat every crawler as a cost. For a B2B brand trying to get found, that framing is backwards. About half the bots in your logs are the reason you appear in search results and AI answers.
So this is a blocking guide with a twist. It's as much about what you keep as what you stop. It works on any platform. If you're on Wix, read this first for the method, then the Wix walkthrough for the exact clicks.
What are the three ways to block bots from a website?
robots.txt asks compliant crawlers to stay out and works for most commercial bots. Server or CDN rules (Apache .htaccess, nginx config, Cloudflare WAF) enforce a block by matching the user agent or IP before the page is served. Bot management products (Cloudflare Bot Fight Mode, Akamai, DataDome) block by behavior rather than identity. Use them in that order, escalating only when the previous layer fails. |
Think of it as a request, then a lock, then a guard.
The request is robots.txt. A text file at your domain root listing which user agents may crawl which paths. Compliance is voluntary, but the crawlers that matter for SEO-data harvesting, MJ12bot, DotBot, PetalBot, AhrefsBot, SemrushBot, BLEXBot and DataForSeoBot, all honor it. So do Googlebot, Bingbot, GPTBot, ClaudeBot, and PerplexityBot. For most sites, this layer alone solves 80% of the problem.
The lock is a rule. On a server you control, that's a rewrite rule in Apache or nginx matching the user agent and returning a 403. Behind a CDN, it's a WAF custom rule doing the same thing at the edge. This stops crawlers that ignore robots.txt, most famously Bytespider. It also stops the polite ones before they cost you a request, which matters at scale.
The guard is behavioral. Bot management products score requests on fingerprint, rate, and pattern rather than declared identity. They're built for credential stuffing, scraping-as-a-service, and DDoS. For a B2B content site, they're usually more than you need and they carry real risk of blocking legitimate integrations and unlisted crawlers. Reach for them last, if at all.
What should the robots.txt look like after you block bad bots?
Add a group below your default rules with one User-agent: line per bot you want to block, followed by a single Disallow: /. Separate groups with a blank line. Leave your User-agent: * group's Allow untouched so every unlisted crawler still has access. Publish and verify at yourdomain.com/robots.txt. |
User-agent: *
Allow: /
User-agent: Bytespider
User-agent: PetalBot
User-agent: MJ12bot
User-agent: DotBot
User-agent: BLEXBot
User-agent: DataForSeoBot
User-agent: SeekportBot
User-agent: serpstatbot
User-agent: Barkrowler
User-agent: ImagesiftBot
Disallow: /
Three rules that keep this from going wrong.
The blank line between groups is load-bearing. Without it, a parser can read the second group as part of the first and apply Disallow: / to every crawler. That's the most common way sites accidentally deindex themselves.
Stacked user agents share one Disallow. You don't need ten blocks.
Anything not named falls through to User-agent: *. That's how the good bots stay allowed without you listing them. If you want to be explicit about the allow side, the robots.txt guide for AI crawlers covers it, and the Wix walkthrough shows the same file in Wix's editor. Drop AhrefsBot and SemrushBot from the block if you use either tool.
Cloudflare's research found only 37% of the top 10,000 domains on its network had a robots.txt file at all in mid 2025. Having one that's correct already puts you ahead.
How do you block bots with server rules?
On Apache, add a RewriteCond matching the user agent and a RewriteRule returning 403 in .htaccess. On nginx, use an if block on $http_user_agent in the server config. Behind a CDN like Cloudflare, create a WAF custom rule with the expression (http.user_agent contains "Bytespider") and action Block. All three stop the request before your page renders. |
Apache (.htaccess):
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Bytespider|PetalBot|MJ12bot|DotBot|BLEXBot) [NC]
RewriteRule .* - [F,L]
nginx:
if ($http_user_agent ~* (Bytespider|PetalBot|MJ12bot|DotBot|BLEXBot)) {
return 403;
}
Cloudflare WAF custom rule:
(http.user_agent contains "Bytespider") or
(http.user_agent contains "PetalBot") or
(http.user_agent contains "MJ12bot")
Action: Block. The full Cloudflare setup, including the three default settings you have to turn off first, is in the Cloudflare guide.
Two cautions. Never match on a short generic string like bot. You'll block Googlebot. And user agents can be faked, so a determined scraper can slip past a user-agent rule by pretending to be Chrome. For SEO-data crawlers and Bytespider, which announce themselves, user-agent rules are sufficient. For adversarial scrapers, you're into behavioral territory.
If you're on a hosted platform with no server access (Wix, Squarespace, Webflow), this layer only exists if you put a CDN in front. The Wix walkthrough covers when that's worth doing and when robots.txt alone is enough.
Which bots should you block?
Block SEO-data crawlers for tools you don't use (MJ12bot, DotBot, BLEXBot, DataForSeoBot, serpstatbot, Barkrowler), search engines outside your market (PetalBot, SeekportBot), image dataset scrapers (ImagesiftBot), and training crawlers that ignore robots.txt and feed no product your buyers use (Bytespider). Keep AhrefsBot and SemrushBot if you use those platforms. |
The test I run for any unfamiliar user agent has three parts, and it's laid out in full in the bad bots hub. Does the operator publish documentation and verifiable IP ranges? Does the crawler honor robots.txt when tested? Does the product it feeds ever show your brand to a buyer? Fail all three, block. Pass the first two and fail the third, judgment call.
Each bot on my block list has its own post explaining the reasoning: Bytespider, PetalBot, MJ12bot, DotBot, BLEXBot and DataForSeoBot. The two I keep, AhrefsBot and SemrushBot, have posts too, because the decision on them depends on your tool stack.
Which bots should you never block?
Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Applebot-Extended, and DuckAssistBot. Each is a distribution path for SEO, AEO, GEO, or LLMO. Blocking any of them removes you from a channel buyers use to find and evaluate vendors. |
This is where B2B customer acquisition starts, and it's the section every other bot-blocking guide leaves out, and it's the one that costs B2B brands the most. Those channels are where B2B customer acquisition starts, and a WAF default doesn't show up on anyone's CAC report.
Cloudflare made AI-crawler blocking the default for new domains on July 1, 2025, and reported 416 billion blocked AI requests in the five months after. Most of the sites behind that number didn't decide to disappear from ChatGPT. A default decided for them. The Cloudflare guide covers turning it off.
The crawlers in that list map to the four engines in the Engine Optimization Matrix. Googlebot and Bingbot are SEO and AEO. The retrieval agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) are GEO. The training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) are LLMO. Cloudflare's data showed training was nearly 80% of all AI crawling by mid 2025, and those models are where buyers now ask for vendor shortlists.
Bingbot deserves special mention. It's the one people block thinking it's harmless. Bing's index is what ChatGPT search retrieves from, so blocking Bingbot cuts a GEO path along with the SEO one. The Bing indexing post explains the chain, and the AI crawlers overview covers every agent in the allow list.
How do you know which bots are hitting your site?
Server access logs are the source of truth. Group requests by user agent and sort by volume. Without log access, Google Search Console and Bing Webmaster Tools show the good-bot side, and a Cloudflare proxy shows everything in its analytics and AI Crawl Control dashboards. GA4 is not useful for this; it filters known bots and most crawlers never execute JavaScript. |
The server log analysis guide walks through pulling and grouping user agents. If your platform doesn't expose logs, Cloudflare's AI Crawl Control is the next best thing, and it includes a robots.txt tab showing which crawlers requested paths you disallowed. That's how you find the ones that need a WAF rule instead of a robots.txt line.
One diagnostic worth knowing. TollBit's Q4 2025 data put robots.txt non-compliance at more than 13% of AI bot requests, up 400% from Q2. If you've added a Disallow and the requests keep coming a month later, you've found one of them.
How do you verify the block worked without breaking anything?
Check the good bots first. Confirm Google Search Console crawl stats and Bing Webmaster Tools crawl volume are unchanged and no pages are newly flagged as blocked by robots.txt. Then confirm the blocked bots stopped, in server logs or Cloudflare Security Events. Repeat quarterly, and after anyone on the IT or security side touches the CDN or WAF. |
The order matters. A blocked bad bot saves you a few requests. A blocked good bot costs you a channel. So verify Google Search Console and Bing Webmaster Tools before you look at anything else. Watch for a jump in crawled but not indexed or robots.txt errors in the Pages report.
Then check the block. Compliant crawlers stop within hours to days. Distributed ones like MJ12bot take up to two weeks. Non-compliant ones like Bytespider don't stop without a server or WAF rule.
And check again quarterly. I've watched a client's citation rate drop for weeks because a security team enabled a bot management default and nobody in marketing knew. Bot blocking is a distribution decision. Treat it like one.
If you want a review of your robots.txt and firewall rules before you change them, book 30 minutes.
Frequently asked questions
How do I block bots from my website?
Start with robots.txt, adding a group that lists each bot to block above a single Disallow rule. For bots that ignore robots.txt, add a server rule (Apache, nginx) or a CDN WAF rule matching the user agent. Verify good bots still crawl after each change.
Does robots.txt actually block bots?
No. It's a request that compliant crawlers honor. Most commercial SEO crawlers and all major search and AI crawlers comply. Bytespider and some scrapers don't, and those need server or WAF enforcement.
How do I block bots in .htaccess?
Add RewriteCond %{HTTP_USER_AGENT} (Bytespider|MJ12bot) [NC] followed by RewriteRule .* - [F,L] with RewriteEngine On. Requests matching those user agents receive a 403.
How do I block bots in nginx?
Add if ($http_user_agent ~* (Bytespider|MJ12bot)) { return 403; } inside the server block. Reload nginx to apply.
Which bots should I block?
SEO-data crawlers for tools you don't use, search engines outside your market, image dataset scrapers, and training crawlers that ignore robots.txt and feed no product your buyers use. Bytespider, PetalBot, MJ12bot, DotBot, and BLEXBot are the usual list.
Which bots should I never block?
Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, and DuckAssistBot.
Should I block AI crawlers?
Not if you want to be cited in ChatGPT, Claude, Perplexity, or Google AI Overviews. Block Bytespider specifically, which feeds no product your buyers use, and leave the rest allowed.
How do I see which bots visit my site?
Server access logs grouped by user agent. Without log access, use Google Search Console and Bing Webmaster Tools for search crawlers and a Cloudflare proxy for everything else. GA4 doesn't show crawlers.
Can I block bots by IP address?
Yes, for crawlers that publish IP ranges. It's unreliable for distributed crawlers like MJ12bot and unnecessary for compliant ones. User-agent matching is simpler and covers most cases.
How often should I review bot blocking rules?
Quarterly, and any time someone changes CDN or WAF settings. New crawlers appear constantly and security defaults can silently block the crawlers you need.
Does blocking bots improve SEO?
Indirectly, by reducing server load and wasted crawl. It doesn't raise rankings directly, and blocking the wrong bots hurts far more than blocking junk helps.
Is blocking bots different on Wix or Squarespace?
The method is the same but you only have robots.txt natively. Server rules require putting a CDN like Cloudflare in front of the site.
