Method and apparatus for score normalization for information retrieval applications

A data processing system for normalizing a score associated with a document, comprising:

• a memory having program instructions;
• and
• a processor responsive to the program instructions to:
• determine statistics relating to scores assigned to a set of training documents not relevant to a topic, the scores representing a measure of relevance to the topic;
• normalize a score assigned to a testing document based on the statistics to obtain a normalized score wherein:
• normalizing adjusts the testing document relevance score based on the statistics to be comparable to other scores from which the statistics were determined, and
• the normalized score is a better predictor of probability of the testing document being relevant than the testing document relevant score;
• compare the normalized score to a threshold score to obtain a comparison;
• and
• designate the testing document as relevant or not relevant to the topic based on the comparison.

A computer-readable medium containing instructions for performing a method for normalizing a score associated with a document, the method facilitated by a human annotator comprising:

• (a) establishing, through the human annotator, a query relevant to a topic (on-topic) and a set of training documents not relevant to the topic (off-topic);
• (b) assigning, through the computer environment, a training document relevance score to each one of the training documents, each training document relevance score representing a measure of relevance of its respective document to the topic;
• (c) determining, through the computer environment, statistics relating to all training document relevance scores;
• (d) receiving a testing document;
• (e) calculating, through the computer environment, a score of relevance of the testing document to the topic to obtain a testing document relevance score;
• (f) normalizing, through the computer environment and based on the statistics, the testing document relevance score to obtain a normalized score wherein:
• normalizing adjusts the testing document relevance score based on the statistics to be comparable to other scores from which the statistics were determined, and
• the normalized score is a better predictor of probability of the testing document being relevant than the testing document relevant score;
• (g) establishing, through the computer environment, a threshold score representing a relevance threshold for the topic;
• (h) comparing the normalized score to the threshold score to obtain a comparison;
• and
• (i) designating the testing document as relevant or not relevant to the topic based on the comparison.

A method, facilitated by a human annotator and performed in a computer environment for normalizing a score associated with a document, the method comprising the steps of:

• (a) receiving through the human annotator a query including a topic and a set of training documents not relevant to the topic (off-topic);
• (b) assigning, through the computer environment, a training document relevance score to each one of the training documents, each training document relevance score representing a measure of relevance of its respective document to the topic;
• (c) determining, through the computer environment, statistics relating to all training document relevance scores;
• (d) receiving a testing document;
• (e) calculating, through the computer environment, a score of relevance of the testing document to the topic to obtain a testing document relevance score;
• and
• (f) normalizing, through the computer environment and based on the statistics, the testing document relevance score to obtain a normalized score wherein:
• normalizing adjusts the testing document relevance score based on the statistics to be comparable to other scores from which the statistics were determined, and
• the normalized score is a better predictor of probability of the testing document being relevant than the testing document relevant score.

A method facilitated by a human annotator and performed by a processor for searching for documents relevant to a topic comprising the steps of:

• the human annotator both sending a query including a topic to the processor and establishing a set of training documents not relevant to the topic (off-topic);
• and
• the human annotator receiving results from the processor indicating a document relevant to the topic, wherein the processor:
• assigns, through the computer environment, a training document relevance score to each one of the training documents, each training document relevance score representing a measure of relevance of its respective document to the topic;
• determines , through the computer environment, statistics relating to all training document relevance scores;
• receives a testing document;
• calculates, through the computer environment, a score of relevance of the testing document to the topic to obtain a testing document relevance score;
• normalizes, through the computer environment and based on the statistics, the testing document relevance score to obtain a normalized score wherein:
• normalizing adjusts the testing document relevance score based on the statistics to be comparable to other scores from which the statistics were determined, and
• the normalized score is a better predictor of probability of the testing document being relevant than the testing document relevant score;
• establishes, through the computer environment, a threshold score representing a relevance threshold for the topic;
• compares the normalized score to the threshold score to obtain a comparison;
• and
• designates the testing document as relevant or not relevant to the topic based on the comparison.