Unsupervised topic modeling for short texts

#The present disclosure relates to the unsupervised modeling and determination of short text topics. More specifically, the disclosure is directed representing topics in a corpus of short texts by creating a Gaussian mixture model from a dense vector distributed representation of each word. The distributed representations are computed by processing the corpus using a sampling window having a length comparable to that of the short texts.

#One of the fundamental problems in natural language processing (NLP) is to learn meaning (at word, phrase, sentence or discourse level). Often, one would like to learn meaning or semantics in a data-driven fashion, possibly in an unsupervised manner. Deriving meaning from linguistic units has immense benefit in tasks such as information retrieval, machine translation (concept-based) and deeper analysis of texts for various business related decision making or troubleshooting. More recently, semantics have become important to glean meaning from Big Data such as customer reviews, tweets, user comments, etc.

#A popular way to infer semantics in an unsupervised manner is to model a document as a mixture of latent topics. Several latent semantic analysis schemes have been used to good success in inferring the high level meaning of documents through a set of representative words (topics). However, the notion of a document has changed immensely over the last decade. Users have embraced new communication and information media such as short messaging service (SMS), Twitter®, Facebook® posts and user comments on news pages/blogs in place of emails and conventional news websites. Document sizes have been reduced from a few hundred words to a few hundred characters while the amount of data has increased exponentially.

#The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

#FIG. 1 is a schematic diagram illustrating obtaining distributed representations of words using a continuous bag-of-words architecture according to embodiments of the disclosure.

#FIG. 2 is a schematic diagram illustrating obtaining distributed representations of words using a deep neural network architecture according to embodiments of the disclosure.

#To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures.

#An unsupervised topic model is disclosed that uses distributed representations of words. The distributed word representations are learned through deep neural networks or linear models represent words in D, where D is the dimensionality of the vector space. In embodiments, the low-dimensional semantic vector space represented by the dense word vectors is modeled using Gaussian mixture models (GMMs). The K components of the Gaussian mixture model can be considered as the latent topics that are captured by the model. For short messages, the distributed representations learned over limited context windows have been found to be sufficient in capturing the distributional similarity of words as well as semantics of the message. In comparison with previous approaches to topic modeling, the distribution over documents is completely ignored and instead the entire corpus is considered, thereby eliminating the need for aggregation over short messages. The disclosed framework presented here is unsupervised, language agnostic and scalable.

#Various methodologies of semantic analysis have been conventionally used on collections of documents that are typically at least a few hundred words in length. With the recent advent of SMS, Twitter® and Facebook®, several efforts have addressed the application of topic models to short texts. For example, tweets from a single user have been aggregated into documents, from which topics are induced. In another example, a supervised version of latent Dirichlet allocation (LDA) was used to model individual messages.