How to Get Google to Crawl Your Robots File
Robots.txt provides direction for search bots, including Google's Googlebot. You create the file in Notepad and upload it to the root of the domain. Google automatically searches for the file, but you must set the robots.txt to allow Googlebot to crawl the site. You can also specify sections of your site that you don't want Googlebot to crawl.
Other People Are Reading
Instructions
-
-
1
Click the Windows "Start" button and type "notepad" in the menu text box. Press "Enter" to open the text editor.
-
2
Type the following text to instruct Googlebot to crawl the site:
User-agent: *
This text instructs all bots to crawl the entire site.
-
-
3
Add any "Disallow" sections in the robots.txt. For instance, if you want to disallow the "private" folder, add the following text to your file:
Disallow: /private/
-
4
Press the "Ctrl" and "S" keys to save the changes. Name the file "robots.txt" and upload it to your website domain. Each domain host offers different methods to upload files, so the upload procedure depends on the permissions and upload applications provided by the host.
-
1
