How to Embed a Web Spider Script
Whenever you start a website, Web spiders (also known as search engine spiders) automatically crawl your site pages. This allows search engines to index your website in the search engine database. The problem with Web spiders is that sometimes the spider will index your site before you are finished designing it. The spider may also add site pages that you do not want to include in search results. The good news is that by embedding a simple HTML script, you can block specific Web pages from spiders.
Other People Are Reading
Instructions
-
-
1
Log in to the “Control Panel” of your website.
-
2
Click “File Directory” to access all of your page files and folders. Write down the name of each folder you want to block from Web spiders. For instance, the images folder, documents folder and home folder.
-
-
3
Click “New” to create a new page file. Name the new page file “robots.txt.” Click “Save.”
-
4
Click the “robots.txt” page file inside the File Directory. Select “Edit” to launch the page's HTML editor.
-
5
Locate the “<head>” and “</head>” tags inside the HTML editor window.
-
6
Paste the following HTML code anywhere between the two head tags. The code instructs Web spiders not to access your images, documents or home folders. Replace the folder names with your own folder names. Feel free to add more folders as needed.
User-agent: *
Disallow: /images
Disallow: /documents
Disallow: /home -
7
Click “Save” to save your page changes.
-
1
