How Do I Unrestrict Robots for Google?
When Google cannot access part of or your entire website, the areas it cannot access will not get indexed in the search results. The robots.txt file is responsible for telling the Google bots what they can and cannot access. Fortunately, removing the blocked files or directories on your website is fairly simple. You can change the robots.txt file by logging into Google Webmaster Tools, generating a new robots.txt file and uploading it to your site.
Other People Are Reading
Instructions
-
-
1
Visit the Google Webmaster Tools website (see Resources). Sign in with your Google account, if you have one. If you do not, click "Create an Account Now."
-
2
Click the gray "Add Site" button if your site is not already listed in Google Webmaster Tools. You'll need to verify you're the owner of your site by following one of the onscreen instructions. Navigate back to the Webmaster Tools website after your site is verified.
-
-
3
Click your site link under "Sites." Click the blue "Site Configuration" link on the left side of the page, under "Dashboard." Click "Crawler Access."
-
4
Click the blue "Generate robots.txt" link in the middle of the page. Click the circle next to "Allow all." If you wish to allow Google to access everything on your website, click "Download" at the bottom of the page. If you want to block Google from crawling certain areas of your website, click the "Action" drop-down menu and select "Block." Click the "User Agent" drop-down box and click which Google bot you want to block; then type the area of your website you wish to block in the text box. Click "Add rule" and then click "Download."
-
5
Upload the downloaded robots.txt file to the root directory of your website with your FTP manager. If you're unsure of what the root directory is, it's typically labeled "Public_Html."
-
1
