SemrushBot 101: What it is and Why you Want to Keep it
Key takeaways
|
I'm biased here and I'll say so up front. MQL Magnet is a Semrush Agency Partner, and I use the platform every day for keyword research, cannibalization checks, and site audits. So when a client asks whether they should block SemrushBot, my first reaction is "please don't."
But the honest answer is more specific than that, because SemrushBot isn't one crawler.
What is SemrushBot?
SemrushBot is the family of web crawlers operated by Semrush, the SEO and marketing intelligence platform. The primary variant builds Semrush's backlink index. Other variants power Site Audit, Brand Monitoring, SEO Writing Assistant, and On Page SEO Checker, and those only crawl sites their owners have configured in a Semrush project. |
Semrush documents the variants on its crawler page. The ones you're most likely to see in logs:
User agent | Job | Crawls uninvited? |
SemrushBot | Backlink index (Backlink Analytics, Authority Score) | Yes |
SemrushBot-BA | Backlink Audit for project owners | No |
SemrushBot-SI | Site Audit for project owners | No |
SemrushBot-SWA | SEO Writing Assistant checks | No |
SemrushBot-CT | Content Analyzer | No |
SemrushBot-BM | Brand Monitoring | Yes |
The distinction matters. The core SemrushBot is like AhrefsBot or DotBot. It maps the web for everyone's benefit. SemrushBot-SI is like Moz's Rogerbot. It only shows up because you set up a project and asked for an audit.
Does SemrushBot respect robots.txt?
Yes. Semrush documents that all its crawlers honor robots.txt Disallow directives and the Crawl-delay directive. Changes to robots.txt are re-read on the next visit, and Semrush notes this can take up to an hour. A named Disallow block is sufficient. No network-edge rule is needed. |
Semrush's crawlers are among the best documented in the industry. They publish the user agent strings, explain each variant's purpose, and describe robots.txt handling in detail. That's the standard I hold every crawler to in the bad bots hub, and SemrushBot clears it easily. Compare Bytespider, which publishes almost nothing and honors less.
Should you block SemrushBot?
If you or your agency use Semrush, block nothing. Blocking the core SemrushBot makes your own domain's Backlink Analytics and Authority Score data incomplete, and blocking SemrushBot-SI breaks your Site Audit. If nobody in your organization uses Semrush, you can block the core crawler. The project-based variants won't visit you anyway. |
This is where the "one decision" framing gets people in trouble. I've seen teams write User-agent: SemrushBot* or a broad WAF rule matching "semrush" in the user agent, then open a ticket asking why Site Audit returns zero pages. The audit crawler matched the rule.
If you use Semrush, the correct number of SemrushBot variants to block is zero. The SEMrush workflow I use for clients depends on Site Audit crawling every page and on Backlink Analytics seeing the site's full link graph. Block either and you're auditing a partial site.
If you don't use Semrush, the calculus flips the same way it does for AhrefsBot. The core crawler's data benefits Semrush customers, which includes competitors and the agencies pitching them. Blocking it doesn't hide your backlinks (those are discovered on the linking sites), but it does stop Semrush from indexing your pages and outbound links. Reasonable people block it. I don't, because I'm the customer.
How do you block SemrushBot?
To block only the shared-index crawler, use User-agent: SemrushBot followed by Disallow: /. robots.txt user agent matching is a prefix match, so this also matches the hyphenated variants. If you want to keep Site Audit working while blocking the index crawler, you can't do that in robots.txt alone. Keep it simple: block all or none. |
User-agent: SemrushBot
Disallow: /
Because of the prefix matching behavior, that block catches SemrushBot-SI and the rest too. If you're a Semrush customer, don't write it. If you're not, it's the right rule. The Wix walkthrough shows where to paste it.
Does SemrushBot affect AI visibility?
No. SemrushBot feeds Semrush's own indexes. It has no relationship with Google, Bing, OpenAI, Anthropic, or Perplexity. Blocking or allowing it doesn't change your rankings, your presence in AI Overviews, or whether generative engines cite you. |
Where Semrush does intersect with AI visibility is as a measurement tool, not a crawler. Its Position Tracking reports flag AI Overview appearances, and I use that data in the AI Overview optimization work and the citation rate analysis I run for clients. None of that depends on SemrushBot crawling your site. It depends on Semrush crawling Google's results.
So the crawler decision is separate from the tool decision. In the Engine Optimization Matrix, SemrushBot sits outside every cell. The crawlers that occupy the Distribution cells for GEO and LLMO are in the AI crawlers overview.
If you're a Semrush user trying to reconcile what the tool shows against what your logs show, book 30 minutes and I'll walk through it.
About the author
Harold Bell is Founder and CEO of MQL Magnet, a B2B content marketing agency serving enterprise technology brands. He's the creator of the Engine Optimization Matrix (EOM), author of Partner Over Product, and a Forbes Communications Council member.
Frequently asked questions
What is SemrushBot?
SemrushBot is the family of web crawlers operated by Semrush. The core variant builds Semrush's backlink index. Other variants power Site Audit, Backlink Audit, Brand Monitoring, and content tools for project owners.
What is SemrushBot-SI?
SemrushBot-SI is the Site Audit crawler. It only crawls sites that have been added to a Semrush project by their owner. Blocking it breaks your own audits.
What is SemrushBot-BA?
SemrushBot-BA is the Backlink Audit crawler, used when a project owner runs a backlink audit on their own domain.
Does SemrushBot obey robots.txt?
Yes. All Semrush crawlers honor Disallow and Crawl-delay directives. Semrush documents that robots.txt changes can take up to an hour to take effect.
Should I block SemrushBot?
Not if anyone on your team or at your agency uses Semrush. Blocking it degrades your own domain's data and can break Site Audit. If nobody uses Semrush, you can block the core crawler.
Does blocking SemrushBot hide my backlinks?
No. Semrush discovers links to your site by crawling the sites that link to you. Blocking it only prevents it from indexing your own pages.
How do I block SemrushBot?
Add User-agent: SemrushBot and Disallow: / as a separate block in robots.txt. Because of prefix matching, this also blocks the hyphenated variants.
Does SemrushBot affect Google rankings?
No. SemrushBot has no connection to Google, Bing, or any AI model.
Why is Site Audit returning zero pages?
Most often because a robots.txt rule or WAF rule is matching "SemrushBot" broadly and blocking SemrushBot-SI. Check both and allow the Semrush user agents.
How does SemrushBot compare to AhrefsBot?
Both are documented, robots.txt-compliant SEO platform crawlers. Semrush splits its crawlers into task-specific variants; Ahrefs uses a single AhrefsBot. The decision to allow either depends on which platform you use.
