How to Stop the Users of Robots in Joomla!
Robots are special programs that access websites to look at their content. They follow links and act just like users in many aspects. Search engines use robots to access websites and use the results to index the website content. You can block friendly robots -- the ones that follow the established rules for website crawling -- by using a Robots.txt file and setting the robots meta tag. Requiring a password for your Joomla! website will block most, if not all, robots.
Other People Are Reading
Instructions
-
Robots' File
-
1
Open a new text file for editing using a plain text editor such as NotePad or WordPad. The editor should not add any formatting to the file.
-
2
Enter the following two lines of code, pressing the "Enter" key after each line:
User-agent: *
Disallow: /
The asterisk tells robots that arrive at the website that the file applies to all robots. The forward slash -- which means the root directory -- tells the robots that the file applies to the entire site.
-
-
3
Save the file and give it the name "robots.txt" using all lowercase letters.
-
4
Upload the file to the root directory of your website using your hosting company's file manager or an FTP program of your choosing. If your Joomla! installation is in a directory other than the root, upload the file to that directory as well.
Set the Robots' Meta Tag
-
5
Download the following file from your website folder using the file manager provided by your hosting company or an FTP program of your choosing:
/libraries/joomla/document/html/html.php
Directories, folders and file names are case-sensitive on some hosting systems.
-
6
Open the file with a plain text editor such as NotePad or WordPad. Press "Control" and "F" to open a search box and type "Robot" in the search text field. Click the "Find Next" button. The editor will find the line that sets the robots' meta tag.
The line looks like this: $this->setMetaData('robots', 'index, follow' );
-
7
Edit the line to match the following code or just copy and paste this line in to replace the original line:
$this->setMetaData('robots', 'noindex, nofollow' );
This tells robots visiting your website not to index it or follow any links.
-
8
Click "File," then "Save" to save the file. Upload the file to your website and place it back in the original directory, overwriting the original file.
Upload to this directory: /libraries/joomla/document/html/
Your website pages will now be rendered with the robots' tag set to the "noindex" and "nofollow" values.
-
1
Tips & Warnings
You can password-protect your website easily and stop noncompliant robots. Set the access level for all the menu items to "Registered" or "Special" and provide a log-in link on the front page. The "Registered" access level will allow anyone who has logged in to see the rest of the site and will exclude robots from entering it.
Always back up your site before you being making changes to it. If something unexpected happens, you can always restore from your backup.
