How to Block Bots with robot.txt File via cPanel?

Bot traffic is non-user traffic that can seriously impair the functionality of your website. Protecting your website against abusive bot traffic is crucial, and there are a few ways to achieve this. This tutorial will teach you how to use your cPanel account’s robot.txt file to stop bots.

Robot.txt is a text file that you may place in the root directory of your website to allow search engine bots to crawl and index it. It seems that it also determines which files the bots may access, thus configuring it is crucial. This is also the reason why you may use the robot.txt file to prevent these bots from crawling your website in search engines.

To Block Bots Using Robot.txt File

1. Log in to your cPanel account.

2. Go to the FILES section and click on File Manager.

The File Manager interface will open. You will have to create the robot.txt file.

3. Click the +File button on the top left corner of the interface.

4. Enter the file name as robot.txt and click the Create new file button.

5. Locate and right-click on the robot.txt file. Select the Edit option.

The file will open for you to enter/edit its information.

6. Copy this code in the file.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
RewriteRule .* – [F,L]

This code will block the bot from the Baidu search engine.

7. Click the Save Changes button and Close the file.

With this robot.txt file, the Baidu search engine cannot use bots on your website.

Note

You can also add the bot crawler of any other search engine to the code preventing it from indexing your website.

In short, you can even block bots from other search engines with the robot.txt file by editing it via cPanel File Manager.