How to Recognize Web Spiders
Web spiders, also known as search engine spiders or crawlers, are automated software programs that examine the contents of websites. Companies like Google and Yahoo! use Web spiders to analyze websites and then list them in their search engines according to the information found. Web spiders can be identified by site administrators because of the way in which they visit sites. While human visitors often browse your site seemingly at random, a Web spider scours the pages in detail.
Other People Are Reading
Instructions
-
-
1
Look at your website's server logs. You may have to contact your hosting provider for access. Every server keeps logs of visitors to the websites it hosts, and site administrators can ask to view those logs.
-
2
Examine the referring URL column of the logs, looking first at any dates on which you believe a search engine spider visited your site. Most human Web visits come from other websites. A Web spider has a blank referring URL.
-
-
3
Examine the Activity column of visits containing a blank URL. Web spiders also leave their mark in how they browse your website. Find the Activity column for the visit and you'll see that the spider visited every page on your website.
-
1
