What Is a Search Engine Bot?

A search engine bot is an automated program used to collect information from websites across the Internet. The search engine then organizes and records the information for use in its search results. Numerous bots "crawl" the Internet by all major search engines at any given time.

Alternate Names

Search engine bots are also commonly known as web crawlers, web spiders, web scutters and web robots.

Basic Function

Bots create copies of all visited pages. The search engine then indexes those copies in their archive, which contributes to faster and more accurate searching.

Policies

A search engine bot is controlled by four policies: which pages to crawl, how often to check for changes to a page, how to crawl pages without overloading the pages' servers, and how to coordinate its jobs with other web crawlers. These can often be controlled by rules found in individual pages or site maps.

Crawl Method

Search engine bots crawl the Internet by starting with a base list of sites to visit. They then identify every hyperlink on those sites and add those links to the list.

Drawbacks

A comprehensive web crawl is essentially impossible because of the Internet's ever-changing nature, the prevalence of sites using features such as JavaScript or Flash that don't provide any text to parse, and the existence of web pages that are not linked by any other page and thus remain undetectable.

References

• Metamend
• The Web Robot Pages