How to Improve Retrieval Performance by Relevance Feedback

Improving retrieval performance can be enhanced through relevance feedback.

Information retrieval systems (IR) work as tools that query input and return information as output. An example of a modern IR is the search engine. The term "relevance feedback" was introduced more than twenty years ago as a way to denote an automatic process. The system developed involved query formulations following initial retrieval operation. Types of relevance feedback can include that which is explicit, implicit, and pseudo or that which can be categorized as blind feedback. Information retrieval systems engage relevance feedback systems in order to use the results from certain queries to assess their overall adequacy.

Instructions

1. Understand the two common IR techniques, used by most modern systems. The first is the stemming process, which involves removing suffixes from base words. For example, "assisting," "assisted" and "assists" would all be referred to as "assist." Systems that use stemming would require that all words be stemmed prior to indexing. The other common IR technique uses commonly found words from documents used in what are termed "stop lists." Words like "and," "a" and "but" usually comprise most of that list.

2. Evaluate and compare IR systems to determine which one would be best for the purpose. Look at precision and recall within a given set of documents retrieved. Ascertain the level of precision percentage. Decide the level of recall by the percentage of relevant documents that were retrieved.

3. Apply other evaluation aspects such as how well the system satisfies the query, how quickly it satisfies the query, what resources it requires and how easy it is for users to obtain answers.

4. Conduct experiments using varied search terms and strings of words into the IR. Make a note of response times and data retrieved. Keep adjusting the search term words throughout the experiment into different IRs. Review the records to ascertain which IRs work best. Try studying and investigating how experienced users adapt search techniques using traditional online search engine retrieval systems. Have more novice users find ways to incorporate new search terms into the same experiment. Include different types of IRs.

5. Create an experiment wherein all user subjects try the exact same retrieval search engine with the same document collection (such as a group of articles from a newspaper). Have users perform the same searches against the same topics in the same time frames.

6. Compare the results from all conducted experiments. Make grids, charts and graphs to display the data found. This will provide a visual tool from which to glean helpful information. Use the data analysis to ascertain how minimally-trained end-users can use a newly-developed baseline system in a reasonably effective fashion. Interpret the relevance feedback in order to increase overall retrieval effectiveness. Opt for an increased user interaction opportunity. Control relevance feedback to make more efficient interactions and maintain or increase search result effectiveness.

Tips & Warnings

• Use a detailed analysis of behavioral data such as user comments, age and gender to make the experiment results richer and more specific.

• It may be necessary to conduct multiple experiments to ascertain actionable results since real world constraints sometimes may exist.

References

• Supporting Interactive Information Retrieval Through Relevance Feedback