How to control search engines and web crawlers using the robots.txt file

You have the ability to designate which parts of your website should be indexed by search engines and web crawlers and which ones they should pass over. This can be accomplished by placing a robots.txt file in your document root directory and adding instructions to it.

Requests are all that the directions you provide in a robots.txt file are. While many web crawlers and the majority of search engines abide by these instructions, they are not required to. Therefore, if you wish to prevent material from being indexed, you should never rely on the robots.txt file.

Using robots.txt directives

The directives used in a robots.txt file are straightforward and easy to understand. The most commonly used directives are User-agent, Disallow, and Crawl-delay. Here are some examples:

Example 1: Instruct all crawlers to access all files

User-agent: *
Disallow:

In this example, any crawler (specified by the User-agent directive and the asterisk wildcard) can access any file on the site.

Example 2: Instruct all crawlers to ignore all files

User-agent: *
Disallow: /

In this example, all crawlers are instructed to ignore all files on the site.

Example 3: Instruct all crawlers to ignore a particular directory

User-agent: *
Disallow: /scripts/

In this example, all crawlers are instructed to ignore the scripts directory.

Example 4: Instruct all crawlers to ignore a particular file

User-agent: *
Disallow: /documents/index.html

In this example, all crawlers are instructed to ignore the documents/index.html directory.

Example 5: Control the crawl interval

User-agent: *
Crawl-delay: 30

In this example, all crawlers are instructed to wait at least 30 seconds between successive requests to the web server.