How to Block Bad Bots on Wix: Tips to Help You Facilitate AI Crawlers
Key takeaways
|
Believe it or not, mqlmagnet.com runs on Wix. That surprises some people who assume an agency selling technical SEO and AI visibility work would be on a headless stack. But Wix is a useful constraint. If I can get a B2B site cited in ChatGPT and Perplexity from a platform with no server access, so can you.
Bot blocking is one of the places that constraint shows. Wix gives you exactly one native tool for it, and it's enough for most of the problem. This post is the exact setup I run, line by line. If you're on a stack with server access, the general bot blocking guide covers .htaccess, nginx, and WAF rules too.
Where is the robots.txt editor in Wix?
In the Wix dashboard, go to SEO and GEO in the left menu, scroll to Tools and settings, and click Robots.txt Editor. Click View File to see the current file, then edit in the text field. Changes publish immediately and are visible at yourdomain.com/robots.txt. |
Wix generates a default robots.txt for every site. Yours probably looks something like this:
User-agent: *
Allow: /
Disallow: /*?lightbox=
Disallow: /learn/hashtags/
Disallow: /payment-request-page
Disallow: /post/indexnow$
Those Disallow lines are Wix's. They keep crawlers out of lightbox URL variants, hashtag pages, and payment pages. Leave every one of them exactly as it is. The /post/indexnow$ line, if you have it, is from Wix's IndexNow integration.
What should the robots.txt look like after you block bad bots?
Keep the default User-agent: * group unchanged at the top. Add a blank line, then a new group listing each bot to block on its own User-agent: line, followed by a single Disallow: /. Add another blank line and keep the Sitemap directive last. The blank lines separate the groups, which is what makes the rules apply to the right crawlers. |
This is the file I run on mqlmagnet.com. Replace the sitemap URL and any Wix-specific Disallow lines with your own.
User-agent: *
Allow: /
Disallow: /*?lightbox=
Disallow: /learn/hashtags/
Disallow: /payment-request-page
Disallow: /post/indexnow$
User-agent: Bytespider
User-agent: PetalBot
User-agent: MJ12bot
User-agent: DotBot
User-agent: BLEXBot
User-agent: DataForSeoBot
User-agent: SeekportBot
User-agent: serpstatbot
User-agent: Barkrowler
User-agent: ImagesiftBot
Disallow: /
Sitemap: https://www.yourdomain.com/sitemap.xml
Three things make this work.
The blank line between groups. robots.txt parses groups separated by blank lines. Without the gap, a crawler could read the second group as a continuation of the first and apply Disallow: / to everyone. That's the single most common way people accidentally deindex a site.
Stacked user agents. Every User-agent: line above a Disallow: inherits that rule. You don't need ten separate blocks. One shared Disallow covers all ten crawlers.
The star group stays open. Anything not named in your block group falls through to User-agent: * and gets Allow: /. Googlebot, Bingbot, GPTBot, ClaudeBot, and every other crawler you didn't list is unaffected. That's the point.
Why each bot is there is covered in the bad bots hub and the individual posts for Bytespider, PetalBot, MJ12bot, DotBot, and BLEXBot and DataForSeoBot. If you use Ahrefs or Semrush, do not add AhrefsBot or SemrushBot to the list.
Which bots must you never block on a Wix site?
Never add Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, or DuckAssistBot to a Disallow group. Each one is a distribution path for SEO, AEO, GEO, or LLMO. Blocking any of them removes you from a channel your buyers use. |
Wix's own help documentation on blocking AI crawlers walks through adding GPTBot and others to robots.txt, and it's technically accurate. For a B2B brand trying to get cited, it's also exactly the wrong thing to do. Wix also notes that blocking AI crawlers may limit its NLWeb agent features, which need structured content access.
The full argument is in the GPTBot and ClaudeBot posts. The short version is that Cloudflare measured training as nearly 80% of all AI crawling in mid 2025, and the models being trained are where your buyers now ask for vendor shortlists. The AI crawlers overview covers each one, and the robots.txt guide for AI crawlers covers the allow-side syntax if you want to be explicit about it.
Bingbot deserves a special mention because it's the one people block thinking it's harmless. Bing's index is what ChatGPT search retrieves from. Block Bingbot and you've cut a GEO path along with the SEO one. The Bing indexing post explains the chain.
Do Wix App Market bot blockers work?
Not for crawlers. Apps like Blockify and similar IP or country blockers run as JavaScript inside the page, so they only act on visitors who execute JavaScript. Server-side crawlers like MJ12bot and Bytespider request the raw HTML and never run the script. These apps are useful for geo-blocking human visitors or stopping referrer spam. They are not crawler controls. |
This comes up constantly. The apps aren't dishonest. They do what they say for the traffic they can see. But a crawler that fetches HTML and moves on never triggers a client-side rule. If your goal is stopping bots in your server logs, skip the App Market and use robots.txt plus, if needed, Cloudflare.
The same logic applies to anything in a Wix HTML Embed. Embedded content renders inside a sandboxed iframe that crawlers don't see, which is why schema goes in Custom Meta Tags and not embeds. Bot blocking follows the same rule. If it's client-side, crawlers don't know it exists.
How do you block bots that ignore robots.txt on Wix?
You can't do it inside Wix. Wix has no server rules, WAF, or user-agent filtering. For crawlers like Bytespider that ignore robots.txt, proxy the site through Cloudflare and write a WAF custom rule matching the user agent. Wix doesn't officially support third-party proxies, so test carefully and keep the DNS records Wix requires intact. |
This is the 20% robots.txt doesn't cover. TollBit's Q4 2025 data put robots.txt non-compliance at more than 13% of AI bot requests, and Bytespider is the headline case. The Cloudflare guide covers the DNS setup, the WAF rule, and the AI crawler setting you have to turn off before it silently blocks everything you just protected.
If you're not seeing Bytespider or another non-compliant crawler in whatever visibility you have, skip Cloudflare. The robots.txt above is enough. Don't add infrastructure to solve a problem you don't have.
How do you verify the block worked?
Open yourdomain.com/robots.txt in a browser and confirm the new group appears exactly as written. Then wait one to two weeks and check Bing Webmaster Tools and Google Search Console crawl stats to confirm Googlebot and Bingbot are still crawling normally. Wix doesn't expose server logs, so blocked-bot confirmation requires Cloudflare analytics or simply trusting compliant crawlers to comply. |
The check that matters most isn't whether the bad bots stopped. It's whether the good ones didn't. After any robots.txt change, confirm in Google Search Console that crawl requests haven't dropped and no pages are newly flagged as blocked by robots.txt. Do the same in Bing Webmaster Tools. If both look normal, you're done.
Run this check quarterly. Wix occasionally updates its default robots.txt, and I've seen the sitemap line move. A quick look at the live file catches drift before it costs you crawl budget or indexing.
If you want me to review your live robots.txt before you publish, book 30 minutes.
Frequently asked questions
Where is the robots.txt editor in Wix?
Dashboard, then SEO and GEO, then scroll to Tools and settings, then Robots.txt Editor. Click View File to see and edit the current file.
Can I block bots on Wix without robots.txt?
Not server-side. Wix has no WAF or user-agent filtering. App Market blockers are client-side JavaScript and don't affect crawlers. For non-compliant bots, you need Cloudflare in front of the site.
Will blocking bots in robots.txt hurt my Wix SEO?
Only if you block the wrong ones. Blocking Googlebot, Bingbot, or their AI counterparts removes you from search and AI answers. Blocking SEO-spam crawlers has no effect on rankings.
Which bots should I never block on Wix?
Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, and DuckAssistBot.
Do Wix bot blocker apps stop crawlers?
No. They run as JavaScript in the page and only affect visitors who execute scripts. Server-side crawlers fetch raw HTML and never trigger them.
How do I block Bytespider on Wix?
Add it to robots.txt for documentation, but it ignores that. Real blocking requires proxying the site through Cloudflare and writing a WAF rule matching the Bytespider user agent.
Why do I need a blank line between robots.txt groups?
Blank lines separate user agent groups. Without one, a crawler may read your block group's Disallow as applying to the User-agent: * group and stop crawling your whole site.
How long until blocked bots stop crawling?
Compliant crawlers stop within hours to days. Distributed crawlers like MJ12bot can take up to two weeks. Non-compliant crawlers like Bytespider never stop without edge enforcement.
Does Wix support Cloudflare?
Wix supports domains connected by pointing DNS records at Wix, and Cloudflare can proxy that. Wix doesn't officially support third-party proxies, so test carefully and keep all Wix-required DNS records intact.
How do I check that Googlebot is still crawling after changes?
Open Google Search Console, go to Settings, then Crawl stats, and confirm request volume hasn't dropped. Check the Pages report for anything newly flagged as blocked by robots.txt. Repeat in Bing Webmaster Tools.
