Federated knowledge distillation on an encoder of a global asr model and/or an encoder of a client asr model

A method implemented by one or more processors, the method comprising:

• distilling information from a client automatic speech recognition (“ASR”) model to a global ASR model, wherein the client ASR model includes a client encoder, a prediction model, and a joint network, wherein the global ASR model includes a global encoder, the prediction model, and the joint network, and wherein distilling information from the client ASR model to the global ASR model comprises:
• processing an instance of audio data capturing a spoken utterance using the client ASR model to generate one or more predicted coefficients corresponding to the spoken utterance;
• generating global output based on processing the one or more predicted coefficients, a mean vector of the global ASR model, a set of principal directions of the global ASR model using the global ASR model, where the mean vector of the global ASR model and the set of principal directions for the global ASR model are generated using principal component analysis (“PCA”);
• generating a loss based on the one or more predicted coefficients, the global loss, the mean vector of the global ASR model, and the set of principal directions of the global ASR model;
• and
• updating one or more portions of the global encoder based on comparing the loss and the one or more predicted coefficients.

A system comprising:

• memory storing instructions;
• one or more processors operable to execute the instructions to:
• distilling information from a global automatic speech recognition (“ASR”) model to generate a client ASR model, wherein the global ASR model includes a global encoder, a prediction model, and a joint network, wherein the client ASR model includes a client encoder, the prediction model, and the joint network, and wherein distilling the global ASR model to generate the client ASR model comprises:
• processing a set of training instances using principal component analysis (“PCA”) to generate (a) a mean vector for the set of training instances and (b) a set of principal directions for the set of training instances;
• for each of a plurality of training instance in the set of training instances and until one or more conditions are satisfied:
• selecting a given training instance, wherein the given training instance includes an instance of audio data capturing a spoken utterance;
• processing the instance of audio data capturing the spoken utterance using the client ASR model to generate one or more predicted coefficients corresponding to the given training instance;
• processing the instance of audio data capturing the spoken utterance using the global ASR model to generate global output;
• generating a loss based on the global output, the one or more coefficients, the mean vector for the set of training instances, and the set of principal directions for the set of training instances;
• and
• updating one or more portions of the client encoder based on comparing the loss and the one or more predicted coefficients.