How to Find Hidden Web Pages

Search engines generally follow hyperlinks in known websites to find other web pages. Information about each reachable page is stored in an index, which is then used to match web pages with our search criteria. Unlinked web pages, text in video or picture form and information in databases are all usually hidden from us unless we know the exact URLs. There are some resources other than the typical search engines that allow you to use keyword searches to find information in the so-called deep web or hidden web.

Things You'll Need

• Narrowed scope
• Formulated search
• Listing of resources
• Bookmarking tool

Instructions

General Search

1. Decide whether your search will involve general or specialized information. Many databases are highly specialized.

2. Include the term "database" along with your search terms when using a traditional search engine or web directory. Use a term for the particular field or subject. In the search results, you should find databases related to that particular field.

3. Reference the "Resources" section of this article for a listing of searchable databases and online archives. Search in multiple locations for best results.

4. Build a collection of bookmarks to alternate search tools. The author's list in the resources section is meant to serve as a starting list.

Narrowed Search

1. Search user-generated content, including blogs. Nowadays, blogs are a top source of breaking news.

2. Use vertical search engines that specialize in a particular topic and deliver targeted results.

3. Take advantage of the wealth of information in social-networking sites.

Tips & Warnings

• Use resources that rely on human harvesters of hyperlinks and documents.

• Install extensions to your web browser that help collect research sources.

References

• Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity
• White Paper: The Deep Web: Surfacing Hidden Value

Resources

• Author's listing of alternate search tools