Language agnostic multilingual end-to-end streaming on-device ASR system

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:

• receiving, as input to a multilingual automated speech recognition (ASR) model, a sequence of acoustic frames characterizing one or more utterances;
• generating, by an encoder of the multilingual ASR model, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames, wherein the encoder comprises a stack of multi-headed attention layers;
• generating, by a prediction network of the multilingual ASR model, at each of the plurality of output steps, a hidden representation based on a sequence of non-blank symbols output by a final softmax layer;
• generating, by a first joint network of the multilingual ASR model, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps;
• predicting, by a second joint network of the multilingual ASR model, an end of utterance (EOU) token at an end of each utterance based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps;
• and
• classifying, by a multilingual endpointer model, each acoustic frame in the sequence of acoustic frames as either speech, initial silence, intermediate silence, or final silence.