SemrushBot is a simple web crawler software application that collects data about websites by crawling them. This data is then organized and can be accessed by Semrush users.
SemrushBot is similar to any other bots like Ahrefsbot or Googlebot. The main thing SemrushBot does is that it crawls websites to find more pieces of information for various purposes.
A lot of people face issues with the Semrush bot like high bandwidth usage and they plan to block it altogether. Should you block SemrushBot on your website? Well, let’s discuss everything in detail and come to a conclusion.
Semrush is an all in one tool that can solve all your SEO related tasks right from content creation to marketing and everything in between
What Is SemrushBot?
SemrushBot is a web robot or software application used by Semrush to collect information regarding a website. After collecting the data, Semrush compiles them together and is made available to Semrush users for their SEO-related uses.
SemrushBot works like any other web robot. It is assigned a task and the bot performs it repeatedly in a concise manner.
During the crawling process, it collects a lot of data and they are stored in the Semrush database.
Let’s take a look at the data collected by Semrushbot.
What Info Does SemrushBot Collect?
Semrush is a digital marketing and SEO toolkit that comes with more than 50 plus tools to help you increase your website’s search engine presence. If you plan to use Semrush, get a 14-day free trial.
In order to optimize your website better for Google, you will need a lot of insightful data to work on.
SemrushBot collects data for its backlink audit tool, keyword research tool, site audit tool, etc.
Here is what the data collected by SemrushBot is used for:
- Link Building Tool – To analyze competitor’s backlink profile and find possible prospects to reach out to
- Site Audit Tool – Analyzes and reports various on page and techincal SEO issues of a website
- Backlink Index – To include new additions to the Semrush’s huge backlink database
- Brand Monitoring – To find brand mentions and search for articles
- SEO Writing Assistant – To check if a URL is accessible
- Semrush Sensor – To find your website’s SERP volatility score.
These are some of the areas where Semrush uses data collected by SemrushBot.
Different SemrushBots
As mentioned earlier, Semrush is an all-in-one SEO toolkit for website owners. It has various tools for performing various tasks. Apart from the generic SemrushBot, Semrush also has a few other bots specific to some tools.
For example, the SiteAduitBot collects data for Semrush’s Site Audit tool and SemrushBot-BA collects data for the Backlink Audit Tool
| SemrushBot | Used By |
| SiteAuditBot | Site Audit Tool |
| SemrushBot-BA | Backlink Audit Tool |
| SemrushBot-SI | On Page SEO Checker Tool |
| SemrushBot-SWA | SWA Tool |
| SemrushBot-CT | Content Analyzer and Post Tracking Tools |
| SemrushBot-BM | Brand Monitoring |
| SplitSignalBot | SplitSignal Tool |
Should You Block SemrushBot?
Now that you know enough about SemrushBot and different types of SemrushBots, the next question that might come to your mind is should I block Semrush bot?
I will help you make a decision on this.
You should only block Semrush’s Bot if you find yourself in either of these two situations.
SemrushBot Using Too Much Resources
If SemrushBot is using too much of your web server resources like the bandwidth then you may think of blocking it completely. Let’s say SemrushBot hits your website multiple times which is resource-intensive. In such situations, it might make your web server slow.
If this is the case then you have two options, either block SemrushBot completely or delay SemrushBot crawling. If you are about to go with the second way then jump to the next section on how to set crawl delay.
You Don’t Use Semrush Tools
If you do not use or plan to use any of Semrush’s SEO tools then you may block it using robots.txt file or .htaccess file.
If you do not rely on Semrush for your SEO-related tasks then it’s okay to block it.
Over time, SemrushBot collects information about your website and it will be made available to Semrush users. If you are sure you won’t use Semrush tools then you may block it.
Also, make sure that you do not plan to use Semrush in the future as well. A lot of Semrush’s index is based on historical data. So, in the future, if you plan to use Semrush, then you will have little to no historical data for your website.
How To Block SemrushBot?
If you made your mind to block SemrushBot completely then let’s see how to do it.
Keep a few things in mind:
- If your website have sub domains then you must place the code on every subdomain’s robots.txt file.
- yourwebsite.com/robots.txt should always return HTTP 200 status code. (Semrush can handle robots.txt file with status code 3xx as well)
Use Robots.txt File To Block SemrushBot
All you have to do is open your robots.txt file using any editor and add the following lines of code to it.
User-agent: SemrushBot
Disallow: /
Now save the file and upload it back.
If you do not want to completely block SemrushBot but only the bot of a specific tool within Semrush then you may do that as well.
To block Site Audit Tool Bot:
User-agent: SiteAuditBot
Disallow: /
To block Backlink Audit Tool Bot:
User-agent: SemrushBot-BA
Disallow: /
To block On-Page SEO Checker Tool Bot:
User-agent: SemrushBot-SI
Disallow: /
To block SWA Tool Bot:
User-agent: SemrushBot-SWA
Disallow: /
To block Content Analyzer and Post Tracking Tool Bot:
User-agent: SemrushBot-CT
Disallow: /
To block Brand Monitoring Tool Bot:
User-agent: SemrushBot-BM
Disallow: /
To block SplitSignal Tool Bot:
User-agent: SplitSignalBot
Disallow: /
Once saved, it may take a couple of hours or at least 100 requests for SemrushBot to find changes made in the robots.txt file and take actions accordingly.
Use .htaccess File To Block SemrushBot
A few people have reported that blocking SemrushBot via the Robots.txt file didn’t work. If that’s the case then you may block SemrushBot at the server level using the .htaccess file.
Copy and paste the following code into the .htaccess file and save it.
RewriteCond %{HTTP_USER_AGENT} \
SemrushBot |\
These lines of code will completely block SemrushBot at the server level. This means, you completely blocked SemrushBot from accessing information from your server.
How To Delay SemrushBot Crawling?
If SemrushBot is too heavy on your server and you want to delay the crawl rate then Semrush provides us with options for that as well.
Semrush allows us to set a maximum delay of up to 10 seconds.
Use the below-given code to set crawl delay for SemrushBot.
User-agent: SemrushBot
Crawl-delay: 10
Any value of more than 10 seconds will be cut down to 10 seconds.
If no Crawl-delay value is specified then Semrush will automatically adjust the crawl rate depending on the server load at that point.
Semrush Bot: Final Words
This is pretty much everything you need to know about SemrushBot. If you do not use Semrush tools or SemrushBot is eating too much of your web server resources then you may block it using robots.txt file or .htaccess file.
Also, keep in mind not to completely block SemrushBot if you plan to use Semrush tools in the future.
