US 6,473,753 B1 · Method and system for calculating term-document importance

Abstract

A weighting system for calculating the term-document importance for each term within each document that is part of a collection of documents (i.e., a corpus). The weighting system calculates the importance of a term within a document based on a computed normalized term frequency and a computed inverse document frequency. The computed normalized term frequency is a function, referred to as the “computed term frequency function” (“A”), of a normalized term frequency. The normalized term frequency is the term frequency, which is the number of times that the term occurs in the document, normalized by the total term frequency of the term within all documents, which is the total number of times that the term occurs in all the documents. The weighting system normalizes the term frequency by dividing the term frequency by a function, referred to as the “normalizing term frequency function” (“Γ”), of the total term frequency. The computed inverse document frequency is a function, referred to as the “computed inverse document frequency function” (“B”) of the inverse document frequency. The weighting system identifies a computed normalized term frequency function A and a computed inverse document frequency function B so that on average the computed normalized term frequency and the computed inverse document frequency contribute equally to the weight of the terms.

Claims27

Dependent claims

A method in a computer system for generating a weight for a term within one of a plurality of documents, the method comprising:

generating a term frequency that represents a number of times that the term occurs in the one document;
generating a total term frequency that represents a total number of times the term occurs in the plurality of documents;
calculating a normalized term frequency by factoring the generated term frequency by a normalizing function of the generated total term frequency, wherein the normalizing function substantially equalizes result of term frequency and reciprocal total term frequency on the weight of the term;
and
combining the calculated normalized term frequency with a document frequency to generate the weight for the term.

2 more elements

A method in a computer system for selecting a formula for weighting terms within one of a plurality of documents, the method comprising:

generating an average term frequency that represents an average of term frequencies for at least one of a plurality of terms within each document, the term frequency being the number of times that a term occurs in a document;
generating an average inverse document frequency that represents an average of inverse document frequencies for the at least one of a plurality of terms, the inverse document frequency of the term being the number of documents divided by the number of documents in which the term occurs;
and
selecting a first function and a second function so that the result of the first function of the generated average term frequency is approximately equal to the result of the second function of the generated average inverse document frequency.

1 more element

Dependencies are inferred from the claim text.

Method and system for calculating term-document importance

Life of this patent

Method and system for calculating term-document importance

Life of this patent

Abstract

Claims27