How to Allow Authentication for User-Agent PHP Bots

Web crawlers, or Web "bots" read Web pages, harvesting links and data to populate search engines. Web marketers often encourage these bots, as they increase the visibility of the site to Web traffic. However, some bots populate other search engines or gather data for malicious or marketing purposes. In these cases, you can use built-in PHP functionality to check a bot's user agent authentication and determine how to handle the crawler.

Things You'll Need

• PHP Interpeter

Instructions

1. Look up the User Agent authentication for a particular bot. For example, the Google crawler is represented by the user agents "Googlebot-2.1," "Googlebot-Mozilla-x.x," or "Google-AdSense-x.x"

2. Call the "http_user_agent" variable in the "$_SYSTEM" global array of the PHP interpreter in your script to determine the user agent of the bot:

   <?php

   $agent = $_SYSTEM['HTTP_USER_AGENT'];

3. Check the bot authentication against a list of bot agents. If the http_user_agent variable matches, allow the bot access:

   if ($agent == "Googlebot-2.1"){
   /*authentication code*/
   }

   ?>

References

• PHP: $_SERVER
• Smart IT Consulting: View a Web Page as 'Googlebot'