# US 12,014,728 B2 · Dynamic combination of acoustic model states > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,014,728 B2 - Document ID: US12014728B2 - Status: Granted - Kind: B2 - Application number: 16/363,705 - Filed: 2019-03-25 - Published: 2020-10-01 - Granted: 2024-06-18 - Earliest claimed: 2019-03-25 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Kshitiz Kumar; Yifan Gong - Web page: https://patentgrep.com/patent/US12014728B2 ## CPC - G10L 15/16: using artificial neural networks - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/20: Ensemble learning - G10L 15/065: Adaptation - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence) - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 25/30: using neural networks - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computer implemented method classifies an input corresponding to multiple different kinds of input. The method includes obtaining a set of features from the input, providing the set of features to multiple different models to generate state predictions, generating a set of state-dependent predicted weights, and combining the state predictions from the multiple models, based on the state-dependent predicted weights for classification of the set of features. ## Claims ### Claim 1 A computer-implemented method for classification of input corresponding to multiple different kinds of input, the method comprising: obtaining a set of features from the input, wherein the input comprises speech and the set of features comprise digital representations of speech; providing the set of features to multiple different acoustic models that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the acoustic models comprise hidden layer outputs; generating a set of state-dependent predicted weights from the hidden layer outputs; combining the state predictions from the multiple acoustic models, based on the state-dependent predicted weights, for classification of the set of features; and providing the combined state predictions to a speech decoder to classify the input as one or more spoken words; wherein the set of state-dependent predicted weights are dynamic weights; and wherein the hidden layer outputs are concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different acoustic models to a concatenation function. ### Claim 2 (depends on 1) The method of claim 1, wherein the multiple acoustic models are speech recognition models independently trained on the same or different kinds of speech, and wherein the different kinds of speech include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech and distant speech. ### Claim 3 (depends on 1) The method of claim 1 wherein the multiple models comprise deep long-term short memory (LSTM) acoustic models. ### Claim 4 (depends on 1) The method of claim 1 wherein generating the set of state dependent predicted weights is performed by a deep learning network trained on a training set of state labeled features. ### Claim 5 (depends on 1) The method of claim 1 wherein the set of predicted weights are time dependent static weights. ### Claim 6 (depends on 1) The method of claim 1, wherein the set of predicted dynamic weights are provided by one of the acoustic models comprising Bidirectional long-term short memory (LSTM), generic Recurrent Neural Networks (RNN), or Convolution Neural Networks (CNN). ### Claim 7 A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method of classifying different kinds of input, the operations comprising: obtaining a set of features from the input, wherein the input comprises speech and the set of features comprise digital representations of speech; providing the set of features to multiple different acoustic models that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the acoustic models comprise hidden layer outputs; generating a set of state-dependent predicted weights from the hidden layer outputs; combining the state predictions from the multiple acoustic models, based on the state-dependent predicted weights for classification of the set of features; and providing the combined state predictions to a speech decoder to classify the input as one or more spoken words; wherein the set of state-dependent predicted weights are dynamic weights; and wherein the hidden layer outputs are concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different acoustic models to a concatenation function. ### Claim 8 (depends on 7) The device of claim 7, wherein the multiple acoustic models are speech recognition models independently trained on the same or different kinds of speech, and wherein the different kinds of speech include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech; and distant speech. ### Claim 9 A device comprising: a processor; and a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising: obtaining a set of features from the input, wherein the input comprises speech and the set of features comprise digital representations of speech; providing the set of features to multiple different acoustic models that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the acoustic models comprise hidden layer outputs; generating a set of state-dependent predicted weights from the hidden layer outputs; combining the state predictions from the multiple acoustic models, based on the state-dependent predicted weights for classification of the set of features; and providing the combined state predictions to a speech decoder to classify the input as one or more spoken words; wherein the set of state-dependent predicted weight are dynamic weights; wherein the hidden layer outputs ae concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different acoustic models to a concatenation function. ### Claim 10 (depends on 9) The device of claim 9, wherein the multiple acoustic models are speech recognition models independently trained on the same or different kinds of speech, and wherein the different kinds of speech include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech and distant speech. ### Claim 11 (depends on 9) The device of claim 9 wherein generating the set of state dependent predicted weights is performed by a trained deep learning network trained on a training set of state labeled features.