How to Block Search Engines in Drupal
Drupal uses a search engine blocking method common to all Web servers to prevent search engines from incorporating the Web pages it serves into a search database. This is commonly used for websites that do not have publicly accessible data, or to prevent search engines from requesting excessive traffic from a website.
Other People Are Reading
Instructions
-
-
1
Locate the "robots.txt" file in the root folder of your Drupal website. The root folder is also the location of your homepage, but its location on the Web server will vary depending upon which operating system you are using to run Drupal.
-
2
Create an empty file named "robots.txt" in your root folder if it does not already exist. Drupal version 5 and later will create a robots.txt file by default, but you can also add this file if it is not already there.
-
-
3
Open the robots.txt file in a text editor.
-
4
Replace the contents of the robots.txt file with the following:
User-agent: *
Disallow: /
This tells all Web search engines to skip all content. User agent means the search engine; the "*" is a wildcard, meaning everyone. The "/" means the root, or "top," of your website. Therefore, by disallowing the root, all content on all pages on the site -- which are hierarchically "lower" than the root -- will be skipped.
-
5
Save the robots.txt file. If your Web server uses a Web page cache, reset it so that the new page is sent to search engines when they request the robots.txt page.
-
1
Tips & Warnings
You can also selectively disallow some search engines (such as allowing Google but blocking Bing), or have the search engines index some pages but not others. See robots.txt documentation for details.
