Conditional maximum likelihood estimation of naïve bayes probability models
AssigneeMicrosoft Corporation
InventorsCiprian Chelba, Alejandro Acero
Life of this patent
Estimating the term…
Abstract
A statistical classifier is constructed by estimating Naïve Bayes classifiers such that the conditional likelihood of class given word sequence is maximized. The classifier is constructed using a rational function growth transform implemented for Naïve Bayes classifiers. The estimation method tunes the model parameters jointly for all classes such that the classifier discriminates between the correct class and the incorrect ones for a given training sentence or utterance. Optional parameter smoothing and/or convergence speedup can be used to improve model performance. The classifier can be integrated into a speech utterance classification system or other natural language processing system.
Claims18
A method of constructing a statistical classifier that classifies sentences having words into classes, comprising:
- receiving, in memory accessible by a computer processor, labeled training data comprising text sentences labeled by class;
- calculating, using the computer processor, an initial class probability parameter θy value for each class y based on the number of training data sentences having the corresponding class label;
- constructing a set of binary valued feature vectors for sentences in the training data, each set of feature vectors corresponding to a class label, each feature vector corresponding to a sentence, each feature corresponding to a word k;
- calculating, using the computer processor, initial word/class probability parameter θky values based on training data feature vectors for each class y;
- calculating, using the computer processor, initial word/class improbability parameter θky values for the training data where θky=1−θky;
- and
- constructing and outputting a statistical classifier of parameterized probability values of all classes jointly given sentence x, and wherein constructing a statistical classifier comprises iteratively re-estimating initial parameters θky, θky without recalculating θy, using the computer processor.
A computer readable storage medium including instructions readable by a computer which, when implemented, cause the computer to construct and output a statistical classifier comprising:
- receiving labeled training data categorized into classes;
- calculating an initial class probability θy for sentences on the training data;
- calculating an initial probability θky based on probability that a word k is in class y;
- calculating an initial improbability θky based on improbability that word k is in class y;
- and
- maximizing conditional class likelihood H over the training data by iteratively recalculating the parameters θky, θky without recalculating θy, over all classes jointly to construct the statistical classifier.
A computer readable storage medium comprising:
- a speech recognizer converting multiple speech signals to text;
- and
- a speech utterance classification engine comprising a Naïve Bayes classifier constructed using conditional maximum likelihood estimations wherein a probability θky that a word k is in class y and an improbability θky are iteratively recalculated, and wherein a class probability θy is not reiteratively recalculated, over all classes jointly.
A computer readable storage medium having stored thereon computer readable instructions which, when read by the computer comprise:
- receiving a natural language input through an application programming interface (API);
- providing the natural language input to one or more natural language processing (NLP) components, including a speech utterance classification engine to perform speech utterance and text classification analysis operations on the natural language input, the speech utterance classification engine comprising a Naïve Bayes classifier constructed using conditional maximum likelihood estimations and rational function growth transforms with smoothing and convergence speed-up wherein a probability θky that a word k is in class y and an improbability θky are iteratively recalculated, and wherein a class probability θy is not reiteratively recalculated, over all classes jointly, the speech utterance and text classification analysis operations selected from a plurality of different possible NLP analysis operations selectable through the API;
- and
- returning analysis results from the named entity operations through the API.
Dependencies are inferred from the claim text.