Method and system for calculating phrase-document importance

A method in a computer system for estimating a total phrase frequency of a phrase, the total phrase frequency indicating a total number of occurrences of the phrase within a plurality of documents, each document having terms, each term having a term frequency for each document, the term frequency for a term indicating a number of occurrences of that term within the document, the phrase having component terms, the method comprising:

• estimating a phrase probability for the phrase, the estimated phrase probability being an estimation of the probability that any phrase in documents that contain each component term of the phrase is the phrase, the estimated phrase probability being derived from the term frequencies of the component terms;
• estimating an average phrase frequency for the phrase by multiplying the estimated phrase probability by an average number of terms per document;
• and
• multiplying the estimated average phrase frequency by an estimated number of documents that contain the phrase to estimate the total phrase frequency.

A method in a computer system for generating a weight for a phrase within one of a plurality of documents, each document having terms, the phrase having component terms, the method comprising:

• estimating a number of the plurality of documents that contain the phrase based on term frequencies of the component terms, a term frequency of a term being a number of occurrences of that term in document;
• estimating a total number of times the phrase occurs in the plurality of documents based on the term frequencies of the component terms;
• and
• combining the estimated number of documents that contain the phrase and the estimated total number of times that the phrase occurs in the plurality of documents to generate the weight for the phrase.

A method in a computer system for estimating a number of a plurality of documents that contain a phrase, each document having terms, the phrase having component terms, the method comprising:

• providing an indication of a number of occurrences of each component term within each document;
• providing an indication of a total number of occurrences of all terms within the plurality of documents;
• calculating a probability that a document contains the phrase based on the number of occurrences of each component term within each document and the total number of occurrences of all terms within the plurality of document;
• and
• multiply the calculated probability by the total number of the plurality of document to estimate that number of documents that contain the phrase.

A method in a computer system for estimating a total number of occurrences of a phrase within a plurality of documents, each document having terms, the phrase having component terms, the method comprising:

• providing an indication of a number of occurrences of each component term within each document;
• providing an indication of a total number of occurrences of all terms within the plurality of documents;
• estimating an average number of occurrences of the phrase in documents that contain the phrase based on the number of occurrences of each component term within each document and the total number of occurrences of all terms with the plurality of document;
• and
• multiplying the estimated average number of occurrences of the phrase by the number of the plurality of documents that contain the phrase to estimate the total number of occurrences of the phrase within the plurality of documents.

A computer system for calculating a total phrase frequency of a phrase, each document having terms, each term having a term frequency for each document, the phrase having component terms, comprising:

• a component for calculating a phrase probability for the phrase, the calculated phrase probability being derived from the term frequencies of the component terms;
• a component for calculating an average phrase frequency for the phrase by multiplying the calculated phrase probability by an average number of terms per document;
• and
• a component for multiplying the calculated average phrase frequency by a calculated number of documents that contain the phrase to calculate the total phrase frequency.

A computer-readable medium containing instructions for causing a computer system to generate a weight for a phrase within one of a plurality of documents, each document having terms, the phrase having component terms, by:

• generating a term frequency that represents the number of occurrences of that term in the plurality of documents;
• estimating a document frequency for the phrase based on an estimated phrase probability of the phrase, the phrase probability being derived from term probabilities of the component terms, the term probability of a component term being a ratio of an average of the generated term frequencies for the component terms per document that contains that component term to an average number of terms per document;
• estimating a total phrase frequency for the phrase based on an average phrase frequency for the phrase times the estimated document frequency for the phrase, the average phrase frequency being derived from the phrase probability of the phrase and the average number of terms per document;
• and
• combining the estimated document frequency with the estimated total phrase frequency to generate the weight of the phrase.

A computer-readable medium containing instructions that cause a computer system to generate a weight for a phrase within one of a plurality of documents, each document having terms, the phrase having component terms, by:

• estimating a number of the plurality of documents that contain the phrase based on term frequencies of the component terms;
• estimating a total number of times the phrase occurs in the plurality of documents based on the term frequencies of the component terms;
• and
• combining the estimated number of documents that contain the phrase and the estimated total number of times that the phrase occurs in the plurality of documents to generate the weight for the phrase.