Bytespider: What it is and Why robots.txt Won't Stop It
Key takeaways
|
What is Bytespider?
Bytespider is the web crawler operated by ByteDance, the Chinese technology company behind TikTok, Douyin, and the Doubao AI assistant. It collects web content for training and updating ByteDance's large language models. It identifies itself with the user agent string Bytespider but, unlike most major crawlers, ByteDance has not historically published verifiable IP ranges or a detailed crawler policy. |
That last sentence is the whole problem. OpenAI publishes GPTBot's IP ranges and honors robots.txt. Anthropic does the same for ClaudeBot. Google, Microsoft, and Perplexity all document their crawlers. ByteDance has been an outlier, with a bare-bones documentation page and behavior that doesn't match it.
Does Bytespider respect robots.txt?
Not reliably. Site operators, hosting providers, and Cloudflare have all reported Bytespider continuing to crawl paths explicitly disallowed in robots.txt. Because robots.txt compliance is voluntary, there's no enforcement mechanism short of blocking the requests before they reach your server. |
This isn't unique to Bytespider, but Bytespider is the most prominent case. TollBit's Q4 2025 data showed more than 13% of AI bot requests ignored robots.txt, a 400% increase from Q2 of that year. Cloudflare's own product documentation now includes a robots.txt violation report specifically because enough crawlers do this to warrant a dashboard.
Every polite crawler in this cluster, PetalBot, MJ12bot, DotBot, AhrefsBot, SemrushBot, can be blocked in robots.txt and will stop. Bytespider is why that's not enough.
How much does Bytespider crawl?
A lot, and unpredictably. Cloudflare reported that Bytespider once reached more than 40% of sites on its network, then dropped to 9.37% of sites by mid 2025 as customers blocked it. Its share of AI crawler request volume fell from 14.1% in July 2024 to 2.4% in July 2025, then rebounded past 10% in May 2026 before sliding again. Plan for spikes. |
The volatility matters for capacity planning. In July 2025, Cloudflare put GPTBot at 11.7% of AI crawl traffic and Bytespider at 2.4%, and it looked like Bytespider was fading. Ten months later it was back in the top five. If you sized rate limits around a "Bytespider is declining" assumption, you got surprised.
For context on the overall scale, Cloudflare Radar data from June 2026 put bots at 57.5% of all HTML requests across its network. Bytespider is a slice of that, but on a content-heavy B2B site it can be a visible one. I've seen it exceed Googlebot's request volume in a given week.
Should you block Bytespider?
Yes. For a B2B technology brand, there is no ByteDance product where being in the training data results in your brand being named to an enterprise buyer. Bytespider ignores robots.txt, publishes no verifiable IP ranges, and crawls aggressively. It fails all three tests for a good bot, and it's the first user agent I add to any Cloudflare WAF rule. |
This is the sharpest contrast in the whole bad bots hub. I argue hard for keeping GPTBot and ClaudeBot allowed, because ChatGPT and Claude are where B2B buyers ask for vendor recommendations. That's the entire premise of LLMO.
Bytespider gets none of that argument. Doubao is a consumer assistant for the Chinese market. TikTok's recommendation and search systems don't surface enterprise software vendors. There's no citation to win.
How do you block Bytespider?
Add it to robots.txt for completeness, but don't rely on it. The effective block is a network-edge rule matching the user agent before the request reaches your host. On Cloudflare, create a WAF custom rule with the expression (http.user_agent contains "Bytespider") and the action Block. Do not use the blanket AI crawler toggle, which also blocks the crawlers you want. |
Two steps.
Step one, robots.txt. Add it anyway. It documents your intent, some Bytespider requests do appear to honor it intermittently, and it costs nothing.
User-agent: Bytespider
Disallow: /
Step two, the edge. On a server you control, an Apache or nginx user-agent rule does it; the general blocking guide has both. On Wix, you can't write server rules, so you need Cloudflare in front of the site. The Cloudflare guide covers the DNS setup. Once the proxy is live, the WAF rule is one line:
(http.user_agent contains "Bytespider")
Action: Block. That's it. I also add the other known robots.txt-ignoring agents to the same rule with or operators as they come up.
What you don't do is flip Cloudflare's AI crawler toggle. It blocks Bytespider, but it also blocks GPTBot, ClaudeBot, PerplexityBot, and the retrieval agents. That's the mistake that costs citation share. Name the bot. Block the bot. Leave the rest alone.
Does blocking Bytespider affect AI visibility?
No, for B2B brands. Bytespider feeds ByteDance models that don't surface enterprise vendors to buyers. Blocking it has no effect on ChatGPT, Claude, Perplexity, Google AI Overviews, or Bing. It's the rare AI crawler you can block without any GEO or LLMO cost. |
In the Engine Optimization Matrix, Bytespider technically sits in the LLMO Distribution cell, but for a market where the models it trains never answer your buyers' questions. That's why I don't lump it with the AI crawlers that matter. The category is the same. The value is not.
If you want help writing a WAF rule that blocks this and nothing else, book 30 minutes.
Frequently asked questions
What is Bytespider?
Bytespider is ByteDance's web crawler. It collects content for training ByteDance's AI models, including those behind the Doubao assistant. ByteDance also owns TikTok.
Does Bytespider obey robots.txt?
Not reliably. Cloudflare, hosting providers, and site operators have repeatedly reported Bytespider crawling disallowed paths. robots.txt alone will not stop it.
How do I block Bytespider?
Add it to robots.txt for documentation, then block it at the network edge. On Cloudflare, a WAF custom rule with the expression (http.user_agent contains "Bytespider") and action Block is sufficient.
Should I block Bytespider?
Yes, for B2B brands. No ByteDance product surfaces enterprise vendors to buyers, it ignores robots.txt, and it crawls aggressively. It fails every test for a good bot.
Does blocking Bytespider hurt my AI visibility?
No. ChatGPT, Claude, Perplexity, Google, and Bing don't use ByteDance's crawl. Blocking Bytespider has no effect on citations in any of them.
How much traffic does Bytespider generate?
It varies widely. Cloudflare measured its share of AI crawl traffic at 14.1% in July 2024, 2.4% in July 2025, and above 10% in May 2026. On content-heavy sites it can exceed Googlebot's request volume in a given week.
Can I block Bytespider on Wix?
Not natively. Wix's robots.txt editor lets you add a Disallow, but Bytespider ignores it. You need Cloudflare proxying the site with a WAF rule.
Why shouldn't I use Cloudflare's AI bot blocking toggle for Bytespider?
Because that toggle also blocks GPTBot, ClaudeBot, PerplexityBot, and the retrieval agents that feed AI citations. Write a specific rule for Bytespider instead.
Does Bytespider publish IP ranges?
ByteDance has not historically published verifiable IP ranges for Bytespider the way OpenAI, Anthropic, and Google do for their crawlers. Block by user agent.
How is Bytespider different from GPTBot?
Both are AI training crawlers. GPTBot honors robots.txt, publishes IP ranges, and feeds ChatGPT, which B2B buyers use for vendor research. Bytespider ignores robots.txt, publishes no verifiable ranges, and feeds products that don't surface B2B brands.
