How to Keep Spiders From Subdomains
Spiders are computer programs that search the Internet for new or updated Web pages. When a spider visits your website, it searches for a robots.txt file with instructions you've set up for which pages to crawl or skip. You can include files, folders or your entire site in the robots.txt file. You can give instructions for subdomains, as well, disallowing spiders from visiting. The robots.txt file needs to be placed in the root folder of the subdomain you want to block.
Other People Are Reading
Instructions
-
-
1
Start your favorite text editor, such as Notepad, TextPad or WordPad.
-
2
Add the following to your page:
User-agent: *
Disallow: /
-
-
3
Select "File" and "Save" from the menu. Name the file "robots.txt" and click "Save."
-
1
