# US 12,118,988 B2 · Transducer-based streaming deliberation for cascaded encoders > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,988 B2 - Document ID: US12118988B2 - Status: Granted - Kind: B2 - Application number: 17/933,307 - Filed: 2022-09-19 - Published: 2023-04-06 - Granted: 2024-10-15 - Earliest claimed: 2021-10-04 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Ke Hu; Tara N. Sainath; Arun Narayanan; Ruoming Pang; Trevor Strohman - Web page: https://patentgrep.com/patent/US12118988B2 ## CPC - G10L 15/16: using artificial neural networks - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G06F 40/126: Character encoding - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 15/083: {Recognition networks (G10L15/142, G10L15/16 take precedence)} - G10L 15/183: using context dependencies, e.g. language models - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06N 3/048: Activation functions - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method includes receiving a sequence of acoustic frames and generating, by a first encoder, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The method also includes generating, by a first pass transducer decoder, a first pass speech recognition hypothesis for a corresponding first higher order feature representation and generating, by a text encoder, a text encoding for a corresponding first pass speech recognition hypothesis. The method also includes generating, by a second encoder, a second higher order feature representation for a corresponding first higher order feature representation. The method also includes generating, by a second pass transducer decoder, a second pass speech recognition hypothesis using a corresponding second higher order feature representation and a corresponding text encoding. ## Claims ### Claim 1 A transducer-based deliberation model for cascaded encoders comprising: a first encoder configured to: receive, as input, a sequence of acoustic frames; and generate, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; a first pass transducer decoder configured to: receive, as input, the first higher order feature representation generated by the first encoder at each of the plurality of output steps; and generate, at each of the plurality of output steps, a first pass speech recognition hypothesis for a corresponding first higher order feature representation; a text encoder configured to: receive, as input, the first pass speech recognition hypothesis generated at each of the plurality of output steps; and generate, at each of the plurality of output steps, a text encoding for a corresponding first pass speech recognition hypothesis; a second encoder configured to: receive, as input, the first higher order feature representation generated by the first encoder at each of the plurality of output steps; and generate, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation; and a second pass transducer decoder configured to: receive, as input, the second higher order feature representation generated by the second encoder at each of the plurality of output steps and the text encoding generated by the text encoder at each of the plurality of output steps; and generate, at each of the plurality of output steps, a second pass speech recognition hypothesis. ### Claim 2 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, further comprising a prediction network shared by the first pass transducer decoder and the second pass transducer decoder, the prediction network configured to: receive, as input, a sequence of non-blank symbols output by a final softmax layer; and generate, at each of the plurality of output steps, a dense representation. ### Claim 3 (depends on 2) The transducer-based deliberation model for cascaded encoders of claim 2, wherein the second pass transducer decoder further comprises a joint network configured to: receive, as input, the dense representation generated by the prediction network at each of the plurality of output steps, the second higher order feature representation generated by the second encoder at each of the plurality of output steps, and the text encoding generated by the text encoder at each of the plurality of output steps; and generate, at each of the plurality of output steps, the second pass speech recognition hypothesis. ### Claim 4 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, wherein the second encoder generates the second higher order feature representation without receiving any of the acoustic frames as input. ### Claim 5 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, wherein the first encoder comprises a causal encoder comprising a stack of multi-headed attention layers. ### Claim 6 (depends on 5) The transducer-based deliberation model for cascaded encoders of claim 5, wherein the stack of multi-headed attention layers comprises a stack of Conformer layers. ### Claim 7 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, wherein the second encoder comprises a non-causal encoder comprising a stack of multi-headed attention layers. ### Claim 8 (depends on 7) The transducer-based deliberation model for cascaded encoders of claim 7, wherein the stack of multi-headed attention layers comprises a stack of Conformer layers. ### Claim 9 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, wherein the second pass transducer decoder trains without using any text-only data. ### Claim 10 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, wherein receiving the text encoding generated by the text encoder at each of the plurality of output steps comprises receiving a partial sequence of the text encoding in a streaming fashion. ### Claim 11 (depends on 10) The transducer-based deliberation model for cascaded encoders of claim 10, wherein the second pass transducer decoder further comprises a joint network configured to determine an attention context vector between the partial sequence of the text encoding and a corresponding portion of the second higher order feature representation. ### Claim 12 (depends on 1) The transducer-based deliberation model for cascaded encoders of claim 1, wherein the first and second pass speech recognition hypotheses each correspond to a partial speech recognition result. ### Claim 13 A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising: receiving a sequence of acoustic frames; generating, by a first encoder, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; generating, by a first pass transducer decoder, at each of the plurality of output steps, a first pass speech recognition hypothesis for a corresponding first higher order feature representation; generating, by a text encoder, at each of the plurality of output steps, a text encoding for a corresponding first pass speech recognition hypothesis; generating, by a second encoder, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation; and generating, by a second pass transducer decoder, at each of the plurality of output steps, a second pass speech recognition hypothesis using a corresponding second higher order feature representation and a corresponding text encoding. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the operations further comprise: generating, by a prediction network based on a sequence of non-blank symbols output by a final softmax layer, at each of the plurality of output steps, a dense representation, wherein the first pass transducer decoder and the second pass transducer decoder share the prediction network. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein the operations further comprise, at each of the plurality of output steps, generating, by a joint network, the second pass speech recognition hypothesis based on the dense representation generated by the prediction network at each of the plurality of output steps, the second higher order feature representation generated by the second encoder at each of the plurality of output steps, and the text encoding generated by the text encoder at each of the plurality of output steps. ### Claim 16 (depends on 13) The computer-implemented method of claim 13, wherein the second encoder generates the second higher order feature representation without receiving any of the acoustic frames as input. ### Claim 17 (depends on 13) The computer-implemented method of claim 13, wherein the first encoder comprises a causal encoder comprising a stack of multi-headed attention layers. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, wherein the stack of multi-headed attention layers comprises a stack of conformer layers. ### Claim 19 (depends on 13) The computer-implemented method of claim 13, wherein the second encoder comprises a non-causal encoder comprising a stack of multi-headed attention layers. ### Claim 20 (depends on 19) The computer-implemented method of claim 19, wherein the stack of multi-headed attention layers comprises a stack of conformer layers. ### Claim 21 (depends on 13) The computer-implemented method of claim 13, wherein the second pass transducer decoder trains without using any text-only data. ### Claim 22 (depends on 13) The computer-implemented method of claim 13, wherein receiving the text encoding comprises receiving a partial sequence of the text encoding in a streaming fashion. ### Claim 23 (depends on 22) The computer-implemented method of claim 22, wherein the operations further comprise determining, by a joint network, an attention context vector between the partial sequence of the text encoding and a corresponding portion of the second higher order feature representation. ### Claim 24 (depends on 13) The computer-implemented method of claim 13, wherein the first and second pass speech recognition hypotheses each correspond to a partial speech recognition result.