How to Build a Vertical Search Engine

A vertical search engine is the latest in Internet search technology. A vertical search engine, while useful in some searches, needs to be programmed correctly to provide the most accurate results for the user. Here's how to build a vertical search engine so your users can get the results they need.

Instructions

1. Choose what you'll be programming in. A Java based crawler is one of the most popular and may be easier to execute. Keep your manual handy so if you get stuck you've got somewhere to turn. You might also choose something like HttpClient if you need more functionality, such as building a job based vertical search engine.

2. Decide on the topic or topics your vertical search engine will be focusing on. For instance, you might want users to be able to search only pages related to child development, or vacationing in south Florida.

3. Write a scoring function that scores each site for your categories using a Java crawler such as Nutch or Heritrix. Use a base of keywords or similar detection base to give each website a score.

4. Set a threshold to show relevant websites. Websites with a certain score for your category are considered relevant and are shown as results. For instance, if your vertical search engine finds 50 or more key words on a site, then it is displayed. Sites with a score below this threshold are not.

5. Determine how you will deal with errors, how many simultaneous threads you'll want and if you want to save page content or just the data.