Joint segmenting and automatic speech recognition

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

• receiving a sequence of acoustic frames characterizing one or more utterances;
• and
• at each of a plurality of output steps:
• generating, by an encoder of a joint segmenting and automated speech recognition (ASR) model, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;
• generating, using a prediction network of a decoder of the joint segmenting and ASR model, a hidden representation based on a sequence of non-blank symbols output by a final softmax layer;
• and
• generating, by the decoder of the joint segmenting and ASR model:
• a probability distribution over possible speech recognition hypotheses;
• and
• an indication of whether the corresponding output step corresponds to an end of speech segment,
• wherein the joint segmenting and ASR model is trained on a set of training samples, each training sample in the set of training samples comprising:
• audio data characterizing a spoken utterance;
• and
• a corresponding transcription of the spoken utterance, the corresponding transcription having an end of speech segment ground truth token inserted into the corresponding transcription automatically based on a set of heuristic-based rules and exceptions applied to the training sample,
• wherein generating the indication of whether the corresponding output step corresponds to the end of speech segment comprises generating, using a first joint network of the decoder, the indication of whether the corresponding output step corresponds to the end of speech segment based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps,
• wherein generating the probability distribution over possible speech recognition hypotheses comprises generating, using a second joint network of the decoder, independently of the indication of whether the corresponding output step corresponds to the end of speech segment generated using the first joint network, the probability distribution over possible speech recognition hypothesis based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps, and
• wherein the final softmax layer is configured to, when the first joint network generates the indication that the corresponding output step corresponds to the end of speech segment, select an output symbol with a highest probability in the probability distribution over possible speech recognition hypotheses as a next non-blank output symbol in the sequence of non-blank output symbols at the corresponding output step.