Document content analysis based on topic modeling
AssigneeAmazon Technologies, Inc.
InventorsWeiwei Cheng, Christopher Gonzales
Life of this patent
Estimating the term…
Abstract
A mechanism for progressive topic modeling is disclosed to facilitate document content analysis. Input documents can be sorted and divided into multiple groups. Topic modeling is performed for each group, where the topic modeling for one group is based on the generated topic model from a previous group, if available. The vocabulary used in the topic modeling process can also be updated for each group of documents. The generated topics can be presented in a user interface to facilitate a user in analyzing the documents. The topic modeling mechanism can also be utilized to enhance a document search experience by generating topics from documents contained in search results and presenting topic words to a user as suggested search terms.
Claims20
A computer-implemented method for content analysis based on progressive topic modeling over documents, the method comprising:
- accessing a plurality of documents;
- ordering the plurality of documents according to an attribute associated with the plurality of documents to form ordered documents;
- dividing the ordered documents into a plurality of groups;
- generating a first topic model by performing topic modeling on a first group of the documents based on an original set of words;
- obtaining a second group of documents;
- updating the original set of words for topic modeling to form an updated set of words, wherein updating the original set of words comprises:
- measuring a frequency of individual words that are contained in the plurality of documents or the second group of documents;
- generating a vocabulary frequency score for individual ones of the words based on the weighted sum of the frequency of a corresponding word;
- removing words from the original set of words that have a vocabulary frequency score less than a pre-determined threshold;
- adding words to the original set of words that have a vocabulary frequency score higher than a pre-determined threshold to form an updated set of words;
- generating an updated topic model by performing topic modeling on a second group of the documents based on the first topic model and the updated set of words;
- and
- causing topics in the updated topic model to be presented in a user interface.
An apparatus, comprising:
- one or more processor;
- and
- one or more non-transitory computer-readable storage media having instructions stored thereupon which are executable by the one or more processors and which, when executed, cause the apparatus to:
- generate a first topic model by performing topic modeling on a plurality of documents based on an original set of words;
- obtain additional documents;
- update the original set of words for the topic modeling to form an updated set of words, wherein updating the original set of words comprises:
- measuring a frequency of individual words that are contained in the plurality of documents or the additional documents;
- generating a vocabulary frequency score for individual ones of the words based on the frequency of a corresponding word;
- removing words in the original set of words that have a vocabulary frequency score less than a pre-determined threshold;
- adding words to the original set of words that have a vocabulary frequency score higher than a pre-determined threshold to form an updated set of words;
- and
- generate an updated topic model by performing topic modeling on the additional documents based on the first topic model and the updated set of words.
A non-transitory computer-readable storage media having instructions stored thereupon which are executable by one or more processors and which, when executed, cause the one or more processors to:
- generate a first topic model by performing topic modeling on a plurality of documents based on an original set of words;
- obtain additional documents;
- update the original set of words for the topic modeling to form an updated set of words, the updating the original set of words comprising:
- measuring a frequency of individual words that are contained in the plurality of documents or the additional documents;
- generating a vocabulary frequency score for individual ones of the words based on the frequency of a corresponding word;
- removing words from the original set of words that have a vocabulary frequency score less than a pre-determined threshold;
- adding words to the original set of words that have a vocabulary frequency score greater than a pre-determined threshold;
- generate an updated topic model by performing topic modeling on the additional documents based on the first topic model and the updated set of words;
- and
- cause a user interface to be generated to present the updated topic model.
Dependencies are inferred from the claim text.