How to Turn Off Robots.txt

Robots.txt controls how search engines crawl your site.

Robots.txt is a file many web designers place on their servers to control how search robots "crawl" their site. A strong robots.txt file can help direct sites, like Google and Bing, to the relevant content and block them from accessing protected database. Enabling robots.txt can take a bit of work, but disabling it is much easier. You can always re-enable it later.

Instructions

1. Log into the server folder containing your robots.txt file.

2. Rename your robots.txt file something else, for instance "robotsdisabled.txt" or "robotsoff.txt." In many server access configurations, you can simply click the file name and wait a couple of seconds for the name to highlight to do so. Anything other than the exact spelling "robots.txt" will turn it off, while keeping the file on your server. You can re-enable it at any point by changing the name back.

3. Save the changes to your server. The next time any search engine crawls your site, it will ignore the disabled file.

References

• The Web Robots Pages: About /Robots.txt