Statistical thesaurus, method of forming same, and use thereof in query expansion in automated text searching

A ranking method for ranking term relationship records to allow a set of related concepts to be derived for query expansion, the ranking method comprising:

• a) assigning term weights for respective query terms;
• b) summing the weights to determine a score;
• c) discovering gaps that occur between scores of records from among a plurality of records that are being scored;
• d) assigning, to the scores of the plurality of records being scored, a cutoff score based on the discovered gaps in the scores;
• and
• e) accepting only records whose scores exceed the cutoff score;
• wherein the assigning step includes:
• d1) assigning the cutoff score so that, in the accepting step, few records are accepted when a large gap in score occurs.

A statistical thesaurus, comprising:

• a plurality of collections of indexed records;
• wherein:
• 1) the records constitute respective documents;
• 2) each document includes plural terms that are grouped by weight into different groups within the document;
• and
• 3) the groups are indexed so as to allow the records to be searched by a conventional text document search and retrieval system so as to perform functions of:
• i) adding records to the indexed records and/or
• ii) forming a list of related concepts for possible inclusion in expansion query terms.

A query expansion method in a text document search and retrieval system using a statistical thesaurus to generate expansion query terms, the method comprising:

• dividing the statistical thesaurus into multiple small physical collections, each physical collection having its own index;
• searching the multiple collections in parallel, using the respective indexes;
• and
• merging the search results to form a list of related concepts to be included in the expansion query terms.