Parameter-efficient model reprogramming for cross-lingual speech recognition

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• obtaining an automatic speech recognition model (ASR) model, the ASR model trained on an initial training dataset to recognize speech in a first language;
• receiving an adaptation training dataset comprising one or more transcribed training utterances in a second language different than the first language, each respective transcribed training utterance in the adaptation training dataset paired with a corresponding transcription;
• integrating the ASR model with an input reprogramming module and a latent reprogramming module;
• and
• adapting the ASR model integrated with the input reprogramming module and the latent reprogramming module to learn how to recognize speech in the second language by training the input reprogramming module and the latent reprogramming module on the adaptation training dataset while parameters of the ASR model are frozen.