How to Customize a Web Bot
The "robots.txt" file is a universal file you place in your website's root folder to customize a Web bot as it crawls your website. You must format the robots.txt file properly or the search engines may not index your site correctly; in some cases, your site might be de-indexed. The robots.txt syntax requires only a few lines of code to customize the bot's interaction with your site. A Web bot crawls your site so your site links appear in search engine results.
Other People Are Reading
Instructions
-
-
1
Click the Windows "Start" button, type "notepad" in the search text box and press "Enter." The text editor in which you can create a robots.txt file opens.
-
2
Set up the user-agent line, which creates the customization for individual bots or all bots. The following code sets the user-agent for all bots:
User-agent: *
-
-
3
Block the folders and files you don't want the bots to crawl or index. For example, the following code blocks the folder "private" from robots:
Disallow: /private/
-
4
Allow the bots to crawl and index the rest of the site. The following code allows bots to crawl the entire site except the folders set up in the "disallow" statement:
Allow: /
-
1
