How to Keep Google’s Spiders from Indexing Your Web Pages

Google’s search engine spiders (a.k.a. crawlers) are always hungry for more information. They scour millions of websites looking for fresh content to add to Google’s index. Many times webmasters actually do not want their content to be added to Google’s index, especially if the content is of a sensitive nature, or contains private information. In the age in which we live, hackers are very keen on how to pull up private documents containing sensitive information using nothing more than Google’s search engine. The last thing any webmaster wants to see is their private web pages showing up on search query results, for everyone on the Web to see. So it is of great benefit for us to know how to keep Google’s hungry spiders (also known as “Googlebots”) from indexing the web pages that we don’t want included in their massive index. This article will teach you a simple tip on how to do just that.

Difficulty: Moderately Easy

Instructions

Things You'll Need:

• Some basic knowledge of HTML
• A Windows-based PC (only because I don't know how to do it with a Mac)

1. You’re going to need to be able to edit the source code of your web page. Usually, your web documents can be opened and edited with a basic text editor, such as Microsoft Notepad. On Windows PC’s (and Mac users please forgive me for not knowing the right command), simply right-click on the HTML document that you want to edit, and click “Open with…”, and then select Microsoft Notepad from the list of program options.

2. Now that your document is open in Notepad, you must locate the “head” tags. What you’re going to do is insert what’s known as a “meta tag” into the code, between the opening and closing "head" tags. If you're familiar with HTML, you know that all attributes in HTML tags are surrounded with a left bracket symbol "<" and a right bracket symbol ">". But, since eHow doesn't allow any kind of HTML code within their articles, I will have to substitute the left and right brackets with asterisks instead. Please keep that in mind as you read this tutorial.

3. Now, back to the opening and closing "head" tags. They will look like this: *head* and */head*. Your objective will be to insert the meta tag in Step 4 anywhere between those two "head" tags.

4. Simply copy and paste the following meta tag into your source code, anywhere between the opening and closing “head” tags: *meta name="robots" content="noindex"* (PLEASE NOTE: Do NOT leave the asterisks in that piece of code, or it won’t work! Remember to switch the asterisks back to left & right brackets!) This meta tag instructs Google’s spiders not to index your web page.

Tips & Warnings

• Although I focused on Google’s spiders in this article, that meta tag actually instructs every web crawler (or “robot”) from all the other different search engines to keep from indexing your web page (i.e., Yahoo, Bing, etc.) as well. If you want to get more specific and only exclude Google, but still allow other spiders to crawl and index your page, simply substitute the word “googlebot” for the word “robots” in the meta tag.

Resources

• Using meta tags to block access to your site (Google Webmaster Help)

[IcyCucky]

on 7/12/2009 Great information to know!