# US 12,223,953 B2 · End-to-end automatic speech recognition system for both conversational and command-and-control speech > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,953 B2 - Document ID: US12223953B2 - Status: Granted - Kind: B2 - Application number: 17/737,587 - Filed: 2022-05-05 - Published: 2023-11-09 - Granted: 2025-02-11 - Earliest claimed: 2022-05-05 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Nuance Communications, Inc. - Inventors: Alejandro Coucheiro Limeres; Junho Park - Web page: https://patentgrep.com/patent/US12223953B2 ## CPC - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G10L 15/16: using artificial neural networks - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A contextual end-to-end automatic speech recognition (ASR) system includes: an audio encoder configured to process input audio signal to produce as output encoded audio signal; a bias encoder configured to produce as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; a transcription token probability prediction network configured to produce as output a probability of a selected transcription token, based at least in part on the output of the bias encoder and the output of the audio encoder; a first attention mechanism configured to receive the at least one bias entry and determine whether the at least one bias entry is suitable to be transcribed at a specific moment of an ongoing transcription; and a second attention mechanism configured to produce prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context. ## Claims ### Claim 1 A contextual end-to-end automatic speech recognition (ASR) system, comprising: an audio encoder processing input audio signal to produce as output encoded audio signal; a bias encoder producing as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; and a transcription token probability prediction network producing as output a probability of a selected transcription token, based at least in part on the output of the bias encoder and the output of the audio encoder. ### Claim 2 (depends on 1) The system according to claim 1, wherein the transcription token probability prediction network comprises a joiner module and a Softmax function layer. ### Claim 3 (depends on 1) The system according to claim 1, wherein the transcription token probability prediction network comprises a decoder module and a Softmax function layer. ### Claim 4 (depends on 2) The system according to claim 2, further comprising: a text predictor producing as output a prediction of a next transcription token, based on a previous transcription token, wherein the output of the text predictor is supplied to the joiner module. ### Claim 5 (depends on 4) The system according to claim 4, further comprising: a first attention mechanism receiving the at least one bias entry and determine whether the at least one bias entry is suitable to be transcribed at a specific moment of an ongoing transcription. ### Claim 6 (depends on 5) The system according to claim 5, further comprising: a second attention mechanism producing prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context. ### Claim 7 (depends on 6) The system according to claim 6, further comprising: a label encoder encoding a current state of a transcription; wherein the second attention mechanism produces the prefix penalties at least in part based on the encoded current state of the transcription. ### Claim 8 (depends on 3) The system according to claim 3, further comprising: a first attention mechanism receiving the at least one bias entry and determine whether the at least one bias entry is suitable to be transcribed at a specific moment of an ongoing transcription. ### Claim 9 (depends on 8) The system according to claim 8, further comprising: a second attention mechanism producing prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context. ### Claim 10 (depends on 9) The system according to claim 9, further comprising: a label encoder encoding a current state of a transcription configured to encode the current state of the transcription; wherein the second attention mechanism produces the prefix penalties at least in part based on the encoded current state of the transcription. ### Claim 11 A method of operating a contextual end-to-end automatic speech recognition (ASR) system, comprising: processing, by an audio encoder, an input audio signal to produce as output encoded audio signal; producing, by a bias encoder, as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; and producing, by a transcription token probability prediction network, as output a probability of a selected transcription token, based at least in part on the output of the bias encoder and the output of the audio encoder. ### Claim 12 (depends on 11) The method according to claim 11, wherein the transcription token probability prediction network comprises a joiner module and a Softmax function layer. ### Claim 13 (depends on 11) The method according to claim 11, wherein the transcription token probability prediction network comprises a decoder module and a Softmax function layer. ### Claim 14 (depends on 12) The method according to claim 12, further comprising: producing, by a text predictor, as output a prediction of a next transcription token, based on a previous transcription token, wherein the output of the text predictor is supplied to the joiner module. ### Claim 15 (depends on 14) The method according to claim 14, further comprising: determining, by a first attention mechanism, whether the at least one bias entry output by the bias encoder is suitable to be transcribed at a specific moment of an ongoing transcription. ### Claim 16 (depends on 15) The method according to claim 15, further comprising: producing, by a second attention mechanism, prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context. ### Claim 17 (depends on 16) The method according to claim 16, further comprising: encoding, by a label encoder, a current state of a transcription; wherein the prefix penalties are produced by the second attention mechanism at least in part based on the encoded current state of the transcription. ### Claim 18 (depends on 13) The method according to claim 13, further comprising: determining, by a first attention mechanism, whether the at least one bias entry output by the bias encoder is suitable to be transcribed at a specific moment of an ongoing transcription. ### Claim 19 (depends on 18) The method according to claim 18, further comprising: producing, by a second attention mechanism, prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context. ### Claim 20 (depends on 19) The method according to claim 19, further comprising: encoding, by a label encoder, a current state of a transcription; wherein the prefix penalties are produced by the second attention mechanism at least in part based on the encoded current state of the transcription.