End-to-end segmentation in a two-pass cascaded encoder automatic speech recognition model

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to execute a user interface and a unified end-to-end segmenter and two-pass automatic speech recognition (ASR) model, the unified end-to-end segmenter and two-pass ASR model comprising:
• a first encoder configured to:
• receive, as input, a sequence of acoustic frames, the sequence of acoustic frames characterizing an utterance;
• and
• generate, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;
• a first decoder configured to:
• receive, as input, the first higher order feature representation generated by the first encoder at each of the plurality of output steps;
• generate, at each of the plurality of output steps:
• a first probability distribution over possible speech recognition hypotheses;
• and
• an indication of whether the corresponding output step corresponds to an end of speech segment;
• and
• at a corresponding output step among the plurality of output steps that corresponds to the end of speech segment, emit an end of speech timestamp;
• a second encoder configured to:
• receive, as input:
• the first higher order feature representation generated by the first encoder at each of the plurality of output steps;
• and
• the end of speech timestamp emitted by the first decoder;
• and
• generate, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation;
• and
• a second decoder configured to:
• receive, as input, the second higher order feature representation generated by the second encoder at each of the plurality of output steps;
• generate, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses;
• and
• finalize a transcription of the utterance based on the second probability distributions over possible speech recognition hypotheses generated at the plurality of output steps,
• wherein the user interface is configured to display the finalized transcription of the utterance on a screen of a user device.