# US 2024/0203409 A1 · Multilingual Re-Scoring Models for Automatic Speech Recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0203409 A1 - Document ID: US20240203409A1 - Status: Published application - Kind: A1 - Application number: 18/589,220 - Filed: 2024-02-27 - Published: 2024-06-20 - Earliest claimed: 2021-03-26 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Neeraj Gaur; Tongzhou Chen; Ehsan Variani; Bhuvana Ramabhadran; Parisa Haghani; Pedro J. Moreno Mengibar - Web page: https://patentgrep.com/patent/US20240203409A1 ## CPC - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems - G10L 15/005: {Language recognition} - G10L 15/16: using artificial neural networks - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method includes receiving a sequence of acoustic frames extracted from audio data corresponding to an utterance. During a first pass, the method includes processing the sequence of acoustic frames to generate N candidate hypotheses for the utterance. During a second pass, and for each candidate hypothesis, the method includes: generating a respective un-normalized likelihood score; generating a respective external language model score; generating a standalone score that models prior statistics of the corresponding candidate hypothesis; and generating a respective overall score for the candidate hypothesis based on the un-normalized likelihood score, the external language model score, and the standalone score. The method also includes selecting the candidate hypothesis having the highest respective overall score from among the N candidate hypotheses as a final transcription of the utterance. ## Claims ### Claim 1 A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving transcribed audio training data comprising training audio data corresponding to an utterance paired with a ground-truth transcription of the utterance; during a first pass, processing, using a speech recognition model, the training audio data to generate N candidate hypotheses for the utterance, each corresponding candidate hypothesis among the N candidate hypotheses having a respective first pass score; during a second pass, for each corresponding candidate hypothesis of the N candidate hypotheses: generating, using a neural network rescoring model, a respective second pass score based on the respective first pass score for the corresponding candidate hypothesis; and applying a Softmax function to a respective negative edit-distance between the corresponding candidate hypothesis and the ground-truth transcription; and optimizing model parameters of the neural network rescoring model based on the Softmax function applied to the respective negative edit-distance between the ground-truth transcription and each corresponding candidate hypothesis among the N candidate hypotheses. ### Claim 2 (depends on 1) The method of claim 1, wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture. ### Claim 3 (depends on 1) The method of claim 1, wherein each corresponding candidate hypothesis among the N candidate hypotheses comprises a respective sequence of word labels, each word label represented by a respective embedding vector. ### Claim 4 (depends on 1) The method of claim 1, wherein each corresponding candidate hypothesis among the N candidate hypotheses comprises a respective sequence of sub-word labels, each sub-word label represented by a respective embedding vector. ### Claim 5 (depends on 1) The method of claim 1, wherein the N candidate hypotheses generated using the speech recognition model during the first pass comprises an N-best list of candidate hypotheses. ### Claim 6 (depends on 1) The method of claim 1, wherein the speech recognition model comprises an encoder-decoder architecture including a conformer encoder having a plurality of conformer layers. ### Claim 7 (depends on 1) The method of claim 1, wherein the speech recognition model comprises an encoder-decoder architecture including a transformer encoder having a plurality of transformer layers. ### Claim 8 (depends on 1) The method of claim 1, wherein the operations further comprise selecting one of the N candidate hypotheses as a final transcription of the utterance based on the respective second pass scores generated for the N candidate hypotheses. ### Claim 9 (depends on 1) The method of claim 1, wherein the neural network rescoring model comprises a language-specific neural network rescoring model. ### Claim 10 (depends on 1) The method of claim 1, wherein the neural network rescoring model comprises a multilingual neural network rescoring model. ### Claim 11 A system comprising: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising: receiving transcribed audio training data comprising training audio data corresponding to an utterance paired with a ground-truth transcription of the utterance; during a first pass, processing, using a speech recognition model, the training audio data to generate N candidate hypotheses for the utterance, each corresponding candidate hypothesis among the N candidate hypotheses having a respective first pass score; during a second pass, for each corresponding candidate hypothesis of the N candidate hypotheses: generating, using a neural network rescoring model, a respective second pass score based on the respective first pass score for the corresponding candidate hypothesis; and applying a Softmax function to a respective negative edit-distance between the corresponding candidate hypothesis and the ground-truth transcription; and optimizing model parameters of the neural network rescoring model based on the Softmax function applied to the respective negative edit-distance between the ground-truth transcription and each corresponding candidate hypothesis among the N candidate hypotheses. ### Claim 12 (depends on 11) The system of claim 11, wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture. ### Claim 13 (depends on 11) The system of claim 11, wherein each corresponding candidate hypothesis among the N candidate hypotheses comprises a respective sequence of word labels, each word label represented by a respective embedding vector. ### Claim 14 (depends on 11) The system of claim 11, wherein each corresponding candidate hypothesis among the N candidate hypotheses comprises a respective sequence of sub-word labels, each sub-word label represented by a respective embedding vector. ### Claim 15 (depends on 11) The system of claim 11, wherein the N candidate hypotheses generated using the speech recognition model during the first pass comprises an N-best list of candidate hypotheses. ### Claim 16 (depends on 11) The system of claim 11, wherein the speech recognition model comprises an encoder-decoder architecture including a conformer encoder having a plurality of conformer layers. ### Claim 17 (depends on 11) The system of claim 11, wherein the speech recognition model comprises an encoder-decoder architecture including a transformer encoder having a plurality of transformer layers. ### Claim 18 (depends on 11) The system of claim 11, wherein the operations further comprise selecting one of the N candidate hypotheses as a final transcription of the utterance based on the respective second pass scores generated for the N candidate hypotheses. ### Claim 19 (depends on 11) The system of claim 11, wherein the neural network rescoring model comprises a language-specific neural network rescoring model. ### Claim 20 (depends on 11) The system of claim 11, wherein the neural network rescoring model comprises a multilingual neural network rescoring model.