Speech recognition systems and methods

A computer-implemented method for training a speech recognition system, comprising, for each training pair of a training set, each training pair comprising a plurality of frames of speech audio and a training sequence of tokens:

• deriving a sequence of tokens based on the plurality of frames of speech audio, comprising:
• initializing an accumulator variable;
• for each of the plurality of frames of speech audio until the accumulator variable exceeds or reaches a threshold:
• calculating, by a self-attention decoder layer of a decoder neural network, a halting probability based on an encoding of the respective frame of speech audio, wherein the self-attention decoder layer comprises a first plurality of weights of a greater plurality of weights;
• adding the halting probability to the accumulator variable;
• in response to the accumulator variable exceeding or reaching the threshold, calculating a context vector based on the calculated halting probabilities and the encodings of the respective frame of the plurality of frames of speech audio;
• performing a decoding step using the context vector to derive a token of the sequence of tokens;
• and
• updating the first plurality of weights based on a difference between the derived sequence of tokens and the training sequence of tokens in accordance with a loss function, wherein
• the context vector is further based on respective encodings and halting probabilities for a preceding one or more frames of speech audio, and
• in response to a count of the preceding one or more frames of speech audio exceeding or reaching another threshold, the calculating the context vector is triggered prior to the accumulator variable exceeding or reaching the threshold, the another threshold being different from the threshold.

A system for performing speech recognition, the system comprising:

• one or more processors and one or more memories, the one or more processors being configured to:
• receive a frame of speech audio,
• encode the frame of speech audio,
• calculate by a decoding process a halting probability based on the encoding of the frame of speech audio,
• add by the decoding process the halting probability to a first accumulator variable,
• in response to the first accumulator variable exceeding or reaching a first threshold, calculate by the decoding process a context vector based on the halting probability and the encoding of the frame of speech audio,
• perform by the decoding process a decoding step using the context vector to derive a token,
• execute a function based on the derived token, wherein the executed function comprises at least one of text output or command performance, wherein the context vector is further based on respective encodings and halting probabilities for a preceding one or more frames of speech audio, and
• in response to a count of the preceding one or more frames of speech audio exceeding or reaching a second threshold, trigger calculating the context vector prior to the first accumulator variable exceeding or reaching the first threshold, the second threshold being different from the first threshold.