# US 9,520,127 B2 · Shared hidden layer combination for speech recognition systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,520,127 B2 - Document ID: US9520127B2 - Status: Granted - Kind: B2 - Application number: 14/265,110 - Filed: 2014-04-29 - Published: 2015-10-29 - Granted: 2016-12-13 - Earliest claimed: 2014-04-29 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Jinyu Li; Jian Xue; Yifan Gong - Web page: https://patentgrep.com/patent/US9520127B2 ## CPC - G10L 15/16: using artificial neural networks - G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems - G10L 25/30: using neural networks - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Providing a framework for merging automatic speech recognition (ASR) systems having a shared deep neural network (DNN) feature transformation is provided. A received utterance may be evaluated to generate a DNN-derived feature from the top hidden layer of a DNN. The top hidden layer output may then be utilized to generate a network including a bottleneck layer and an output layer. Weights representing a feature dimension reduction may then be extracted between the top hidden layer and the bottleneck layer. Scores may then be generated and combined to merge the ASR systems which share the DNN feature transformation. ## Claims ### Claim 1 A method of providing a framework for merging two or more automatic speech recognition (ASR) system having a shared deep neural network (DNN) feature transformation, comprising: receiving, by a computing device, at least one utterance; training, by the computing device, the at least one utterance using a DNN feature transformation with a criterion, wherein the DNN feature transformation comprising a plurality of hidden layers; generating, by the computing device, an output from a top hidden layer in the plurality of hidden layers for the at least one utterance; utilizing, by the computing device, the top hidden later output to generate a network comprising a bottleneck layer and an output layer; extracting, by the computing device, one or more weights between the top hidden layer and the bottleneck layer, the one or more weights representing a feature dimension reduction; generating, by the computing device, a first score from a first ASR system based on application of the feature dimension reduction to a model of the first ASR system and generating a second score from a second ASR system based on application of the feature dimension reduction to a model of the second ASR; combining, by the computing device, the first score and the second score to merge the first ASR system and the second ASR system to create a merged system; and training, for the merged system, senone coefficient data for evaluation of spoken utterances. ### Claim 2 (depends on 1) The method of claim 1, further comprising receiving a spoken utterance, and executing ASR recognition for the spoken utterance using the merged system. ### Claim 3 (depends on 2) The method of claim 2, wherein the senone coefficient data is used to evaluate the spoken utterance to determine ASR results. ### Claim 4 (depends on 1) The method of claim 1, wherein receiving, by a computing device, at least one utterance comprises receiving a plurality of training utterances for speech recognition. ### Claim 5 (depends on 1) The method of claim 1, wherein the training of the at least one utterance comprises: training the first ASR system with a cross entropy criterion, the first ASR system comprising a DNN system; and deriving the DNN feature transformation from a top hidden layer of the DNN system. ### Claim 6 (depends on 1) The method of claim 1, wherein the training of the at least one utterance comprises: training the first ASR system with sequential training criterion, the first ASR system comprising a DNN system; and deriving the DNN feature transformation from a top hidden layer of the DNN system. ### Claim 7 (depends on 1) The method of claim 1, wherein utilizing, by the computing device, the top hidden layer output to generate a network comprising a bottleneck layer and an output layer comprises generating a network comprising a low dimension bottleneck hidden layer and a plurality of senones. ### Claim 8 (depends on 1) The method of claim 1, wherein generating, by the computing device, the first score and the second score comprises generating log likelihood scores from a Context Dependent Deep Neural Network-Hidden Markov Model (CD-DNN-HMM) system and a Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) system. ### Claim 9 (depends on 1) The limitation of claim 1, wherein combining, by the computing device, the first score and the second score comprises performing a linear combination of the first score from the first ASR system and the second score from the second ASR system. ### Claim 10 (depends on 1) The method of claim 1, wherein combining, by the computing device, the first score and the second score comprises performing a non-linear combination of the first score from the first ASR system and the second score from the second ASR system. ### Claim 11 A system comprising: at least one processor; and a memory operatively connected with the at least one processor, wherein the memory stores computer-executable instructions that, when executed by the at least one processor, causes the at least one processor to execute a method that comprises: receiving, by a computing device, at least one utterance; training, by the computing device, the at least one utterance using a DNN feature transformation with a criterion, wherein the DNN feature transformation comprises a plurality of hidden layers; generating, by the computing device, an output from a top hidden layer in the plurality of hidden layers for the at least one utterance; utilizing, by the computing device, the top hidden layer output to generate a network comprising a bottleneck layer and an output layer; extracting, by the computing device, one or more weights between the top hidden layer and the bottleneck layer, the one or more weights representing a feature dimension reduction; generating, by the computing device, a first score from a first ASR system based on application of the feature dimension reduction to a model of the first ASR system and generating a second score from a second ASR system based on application of the feature dimension reduction to a model of the second ASR system; combining, by the computing device, the first score and the second score to merge the first ASR system and the second ASR system to create a merged system; and training, for the merged system, senone coefficient data for evaluation of spoken utterances. ### Claim 12 (depends on 11) The system according to claim 11, wherein the method, executed by the at least one processor, further comprises receiving a spoken utterance, and executing ASR recognition for the spoken utterance using the merged system. ### Claim 13 (depends on 11) The system according to claim 11, wherein the training of the at least one utterance comprises: training the first ASR system with at least one of a cross entropy criterion and a sequential training criterion, and deriving the DNN feature transformation from a top hidden layer of a DNN system. ### Claim 14 (depends on 11) The system according to claim 11, wherein the generating of the first score and the second score further comprises generating log likelihood scores from a Context Dependent Deep Neural Network-Hidden Markov Model (CD-DNN-HMM) system and a Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) system. ### Claim 15 (depends on 11) The system according to claim 11, wherein the combining of the first score and the second score occurs by executing at least one selected from a group consisting of: performing a non-linear combination of the first score and the second score, and performing a linear combination of the first score and the second score. ### Claim 16 A computer-readable storage device storing computer executable instructions which, when executed by a computer, cause computer to perform a method of providing a framework for merging systems having a shared deep neural network (DNN) feature transformation, the method comprising: receiving a plurality of training utterances for speech recognition; training a first system with one or more of a cross entropy criterion and a sequential training criterion utilizing the plurality of training utterances, the DNN feature transformation comprising a plurality of hidden layers; generating an output from a top hidden layer in the plurality of hidden layers for the plurality of training utterances; utilizing the top hidden layer output to generate a network comprising a low dimension bottleneck hidden layer and a plurality of senones; extracting one or more weights between the top hidden layer and the low dimension hidden bottleneck layer, the one or more weights representing a feature dimension reduction; utilizing the feature dimension reduction to train a model for a second system following the extraction of the one or more weights between the top hidden layer and the low dimension bottleneck hidden layer; generating a first log likelihood score from the first system based on application of the feature dimension reduction to the first system and a second log likelihood score from the second system based on application of the feature dimension to the model of the second system; combining the first log likelihood score and the second log likelihood score to create a merged system from the first system and the second system, wherein the first system and the second system share the DNN feature transformation; and training senone dependent combination coefficients from the merged system with the one or more of the cross entropy criterion and the sequential training criterion. ### Claim 17 The computer-readable storage medium, wherein the method further comprising receiving a spoken utterance, processing the spoken utterance using the senone dependent combination coefficients, and outputting automatic speech recognition (ASR) results data based on the processing of the spoken utterances. ### Claim 18 (depends on 16) The computer-readable storage device of claim 16, wherein the first system is a Context Dependent Deep Neural Network-Hidden Markov Model (CD-DNN-HMM) system and the second system is a Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) system. ### Claim 19 (depends on 16) The computer-readable storage device of claim 16, wherein combining of the first log likelihood score and the second log likelihood score comprises performing a linear combination of the first log likelihood score from the first system and the second log likelihood score from the second system. ### Claim 20 (depends on 16) The computer-readable storage device of claim 16, wherein combining of the first log likelihood score and the second log likelihood score comprises performing a non-linear combination of the first log likelihood score from the first system and the second log likelihood score from the second system.