How to Stop a Bot From Crawling a Forum
If you own an Internet forum, you might not want search engine bots, or other bots, to crawl your forum. If you exclude a search engine bot, the forum will not appear in the search engine that uses that bot. Other bots sometimes crawl forums to scrap content and forum owners sometimes choose to block these bots. Most bots follow a standard protocol to know whether or not they can crawl a website. Site owners place this protocol in a "robots.txt" file, which is the first thing most bots look for when they reach a site.
Other People Are Reading
Instructions
-
-
1
Type "yoursiteurl.com/robots.txt" into your browser's address bar and press "Enter." If you have a file there, it will display in your browser. Connect to your site via FTP and download a copy of the "robots.txt" file to your computer. Open it in a text editor. If you do not have a file at that address, open Notepad, or other text editor, and create a new file.
-
2
Type "User-agent: name of bot to stop" and press "Enter." To stop all bots from crawling the forum, type an asterisk instead of the name of a particular bot.
-
-
3
Type "Disallow: /forum/" replacing "forum" with the actual directory of your forum on your site. This tells the particular bot not to crawl anything in that directory.
-
4
Save the file. If it is a new file, name it "robots.txt." Reconnect to your site with an FTP client and upload the robots.txt file to the top level directory on your site. If prompted, agree to overwrite the pre-existing robots.txt file.
-
1
Tips & Warnings
Depending on your hosting provider, you might be able to use a file manager and text editor on the server to modify or create a robots.txt file rather than using an FTP client and an offline text editor.
Some malicious bots ignore the robots.txt file. To stop these bots from crawling your forum, ban the IP address they use.
