# US 12,002,450 B2 · Speech recognition systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,450 B2
- Document ID: US12002450B2
- Status: Granted
- Kind: B2
- Application number: 17/183,743
- Filed: 2021-02-24
- Published: 2022-05-19
- Granted: 2024-06-04
- Earliest claimed: 2020-11-16
- Assignees: Kabushiki Kaisha Toshiba
- Applicants: Kabushiki Kaisha Toshiba
- Inventors: Mohan Li; Tudor-Catalin Zorila; Rama Sanand Doddipatla
- Web page: https://patentgrep.com/patent/US12002450B2

## CPC

- G10L 15/00: Speech recognition (G10L17/00 takes precedence)
- G10L 25/87: Detection of discrete points within a voice signal
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 19/02: using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/16: using artificial neural networks
- G10L 2025/783: {based on threshold decision}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for speech recognition, comprising receiving a frame of speech audio; encoding the frame of speech audio; calculating a halting probability based on the frame of speech audio; adding the halting probability to a first accumulator variable; in response to the first accumulator variable exceeding or reaching a first threshold, calculating a context vector based on the halting probability and the encoding of the frame of speech audio; performing a decoding step using the context vector to derive a token; and executing a function based on the derived token, wherein the executed function comprises at least one of text output or command performance.

## Claims

### Claim 1

A computer-implemented method for speech recognition, comprising:
receiving a frame of speech audio;
encoding by an encoder neural network the frame of speech audio, at least one of the encoder neural network or a decoder neural network being used for the speech recognition;
calculating by the decoder neural network a halting probability based on the encoding of the frame of speech audio;
adding by the decoder neural network the halting probability to a first accumulator variable;
in response to the first accumulator variable exceeding or reaching a first threshold, calculating by the decoder neural network a context vector based on the halting probability and the encoding of the frame of speech audio;
performing by the decoder neural network a decoding step using the context vector to derive a token;
executing a function based on the derived token, wherein the executed function comprises at least one of text output or command performance, wherein the context vector is further based on respective encodings and halting probabilities for a preceding one or more frames of speech audio; and
in response to a count of the preceding one or more frames of speech audio exceeding or reaching a second threshold, triggering the calculating the context vector prior to the first accumulator variable exceeding or reaching the first threshold, the second threshold being different from the first threshold.

### Claim 2 (depends on 1)

The method of claim 1, wherein the halting probability and the context vector are calculated using a self-attention decoder layer of a decoder neural network.

### Claim 3 (depends on 2)

The method of claim 2, wherein the self-attention decoder layer is a multi-head self-attention decoder layer comprising a plurality of attention heads, and wherein the halting probability and the context vector are calculated using an attention head of the plurality of attention heads.

### Claim 4 (depends on 3)

The method of claim 3, wherein, for each of the other attention heads of the plurality of attention heads, a respective halting probability is calculated.

### Claim 5 (depends on 4)

The method of claim 4, comprising, for each of the other attention heads of the plurality of attention heads:
adding the respective halting probability to the first accumulator variable; and
in response to the first accumulator variable exceeding or reaching the first threshold, calculating a context vector based on the halting probability and the encoding of the frame of speech audio.

### Claim 6 (depends on 4)

The method of claim 4, wherein there is a respective accumulator variable for each of the other attention heads, and comprising, for each of the other attention heads of the plurality of attention heads:
adding the respective halting probability to the respective accumulator variable;
in response to the respective accumulator variable exceeding or reaching the first threshold, calculating a context vector based on the respective halting probability and the encoding of the frame of speech audio.

### Claim 7 (depends on 6)

The method of claim 6, wherein performing the decoding step is in response to accumulator variables for a final one or more attention heads of the plurality of attention heads exceeding or reaching the first threshold, wherein a remainder of the one or more attention heads have previously exceeded the first threshold.

### Claim 8 (depends on 5)

The method of claim 5, comprising calculating a combined context vector based on the context vectors calculated for each of the plurality of attention heads.

### Claim 9 (depends on 8)

The method of claim 8, wherein the decoder neural network comprises one or more further self-attention decoder layers, and comprising calculating a respective combined context vector using each of the one or more further self-attention decoder layers.

### Claim 10 (depends on 1)

The method of claim 1, wherein the calculation of the halting probability is further based on a preceding context vector.

### Claim 11 (depends on 1)

The method of claim 1, wherein the encoding of the frame of speech audio is by an encoder neural network comprising one or more self-attention encoder layers.

### Claim 12 (depends on 1)

The method of claim 1, wherein the context vector is calculated without trimming the halting probabilities.

### Claim 13 (depends on 1)

A computer program stored on a non-transitory computer readable medium, which, when the computer program is executed by a computer, causes the computer to carry out the method of claim 1.

### Claim 14

A computer-implemented method for training a speech recognition system, comprising, for each training pair of a training set, each training pair comprising a plurality of frames of speech audio and a training sequence of tokens:
deriving a sequence of tokens based on the plurality of frames of speech audio, comprising:
initializing an accumulator variable;
for each of the plurality of frames of speech audio until the accumulator variable exceeds or reaches a threshold:
calculating, by a self-attention decoder layer of a decoder neural network, a halting probability based on an encoding of the respective frame of speech audio, wherein the self-attention decoder layer comprises a first plurality of weights of a greater plurality of weights;
adding the halting probability to the accumulator variable;
in response to the accumulator variable exceeding or reaching the threshold, calculating a context vector based on the calculated halting probabilities and the encodings of the respective frame of the plurality of frames of speech audio;
performing a decoding step using the context vector to derive a token of the sequence of tokens; and
updating the first plurality of weights based on a difference between the derived sequence of tokens and the training sequence of tokens in accordance with a loss function, wherein
the context vector is further based on respective encodings and halting probabilities for a preceding one or more frames of speech audio, and
in response to a count of the preceding one or more frames of speech audio exceeding or reaching another threshold, the calculating the context vector is triggered prior to the accumulator variable exceeding or reaching the threshold, the another threshold being different from the threshold.

### Claim 15 (depends on 14)

The method of claim 14, wherein the context vector is calculated without trimming the halting probabilities.

### Claim 16 (depends on 14)

The method of claim 14, comprising:
encoding, by an encoder neural network, the respective frames of the plurality of frames of speech audio, wherein the encoder neural network comprises a second plurality of weights of the greater plurality of weights;
updating the second plurality of weights based on the difference between the derived sequence of tokens and the training sequence of tokens in accordance with a loss function.

### Claim 17 (depends on 15)

The method of claim 15, wherein the self-attention decoder layer is a multi-head self-attention decoder layer comprising a plurality of attention heads, and wherein the halting probability and the context vector are calculated using an attention head of the plurality of attention heads.

### Claim 18

A system for performing speech recognition, the system comprising:
one or more processors and one or more memories, the one or more processors being configured to:
receive a frame of speech audio,
encode the frame of speech audio,
calculate by a decoding process a halting probability based on the encoding of the frame of speech audio,
add by the decoding process the halting probability to a first accumulator variable,
in response to the first accumulator variable exceeding or reaching a first threshold, calculate by the decoding process a context vector based on the halting probability and the encoding of the frame of speech audio,
perform by the decoding process a decoding step using the context vector to derive a token,
execute a function based on the derived token, wherein the executed function comprises at least one of text output or command performance, wherein the context vector is further based on respective encodings and halting probabilities for a preceding one or more frames of speech audio, and
in response to a count of the preceding one or more frames of speech audio exceeding or reaching a second threshold, trigger calculating the context vector prior to the first accumulator variable exceeding or reaching the first threshold, the second threshold being different from the first threshold.