Highly scalable memory-efficient parallel LDA in a shared-nothing MPP database
AssigneeEMC Corporation
InventorsShengwen Yang, Caleb E. Welton, Guangxin Yang
Life of this patent
Estimating the term…
Abstract
Latent Dirichlet allocation (LDA) analysis on a dataset is performed on an MPP relational database by distributing subsets of said dataset to a plurality of segments of the MPP database, and performing LDA analysis in parallel on the respective subsets on the plurality of segments using Gibbs sampling. An object library on each segment provides executable objects of user defined functions that can be called by an SQL query when the query requires functionality provided by an object.
Claims17
A method of parallel latent Dirichlet allocation (LDA) processing of a dataset in an MPP relational database comprising a master node connected via a network to a plurality of distributed database segments, the dataset comprising a plurality of documents, each document being a mixture of topics where each topic is characterized by a probability distribution over a vocabulary of words, the method comprising:
- distributing subsets of said documents of said dataset to said plurality of segments, where each one of said documents of a subset is distributed to and stored on one single segment of said relational database;
- determining in parallel on each segment for each document of the subset on said segment a per-word and a per-document topic count;
- performing in parallel on each segment LDA analysis using said determined topic counts to assign words to a new topic;
- calculating in parallel on each segment per-word and per-document probability distributions of topic assignments;
- iterating said determining, said performing, and said calculating steps on each segment until a predetermined stop condition is satisfied, and storing results locally in memory on each segment, said iterating comprising updating topic assignments at each iteration by providing on each segment two work tables for holding topic assignments that alternate roles as an input table and as an output table, a first work table storing topic assignments from a previous iteration and serving as an input table of topic assignments to a next iteration, and a second work table serving as an output table for updated topic assignments from said next iteration, and alternating roles of said first and second tables for input and output on each said iteration for updating topic assignments;
- and
- aggregating results from said plurality of segments to form an inference model.
Non-transitory computer readable media for storing executable instructions for controlling the operation of computers on a plurality of segments of an MPP relational database for latent Dirichlet allocation (LDA) processing on said segments in parallel a subset of a dataset, the dataset comprising a plurality of documents, each document being a mixture of topics where each topic is characterized by a probability distribution over a vocabulary of words, said LDA processing comprising:
- distributing subsets of said documents of said dataset to said plurality of segments, where each one of said documents of a subset is distributed to and stored on one single segment of said relational database;
- determining in parallel on each segment for each document of the subset on said segment a per-word and a per-document topic count;
- performing in parallel on each segment LDA analysis using said determined topic counts determine to assign words to a new topic;
- calculating in parallel on each segment per-word and per-document probability distributions of topic assignments;
- iterating said determining, said performing, and said calculating steps on each segment until a predetermined stop condition is satisfied, and storing results locally in memory on each segment, said iterating comprises updating topic assignments of words at each iteration by providing on each segment two work tables for holding topic assignments that alternate roles as an input table and as an output table, a first work table storing topic assignments from a previous iteration and serving as an input table of topic assignments to a next iteration, and a second work table serving as an output table for updated topic assignments from said next iteration, and alternating roles of said first and second tables for input and output on each said iteration for updating topic assignments;
- and
- aggregating results from said plurality of segments to form an inference model.
Dependencies are inferred from the claim text.