Measuring informative content of words in documents in a document collection relative to a probability function including a concavity control parameter
AssigneeXerox Corporation
InventorStephane Clinchant
Life of this patent
Estimating the term…
Abstract
Processing methods and systems are provided for representing documents relative to importance of words in the document. A processor comprising a weighting model of word importance in a document in a collection relative to an importance of the word in other documents in the collection computes a deviation of distribution of the word from a probability distribution of the word in other documents in the collection, where the deviation distribution is weighted in accordance with a concavity control function. A concavity control parameter is adjustable relative to word frequency.
Claims12
A processing system for representing documents relative to importance of words in the document, including:
- a processor comprising a weighting model of word importance in a document in a collection relative to an importance of the word in other documents in the collection by computing a deviation of distribution of the word from a probability distribution of the word in other documents in the collection, wherein the deviation distribution is weighted in accordance with a concavity function.
A method for document representation by measuring informative content of words, comprising:
- associating a probability distribution for a plurality of words in a document collection;
- forming a weighting model of word importance to the collection from the probability distribution therein;
- determining a distribution of a selected word in a particular document in the collection;
- computing a deviation of distribution of the selected word in the document from the probability distribution of the word in the collection in accordance with a concavity function;
- and
- assigning a greater importance of the selected word to the informative content of the document by the greater than computed deviation in accordance with a concavity function whereby the document can be represented relative to the collection by the assigned importance of a plurality of selected words.
Dependencies are inferred from the claim text.