Multilingual and code-switching asr using large language model generated text

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a textual prompt in a first language;
• obtaining a fine-tuned prompt embedding configured to guide a large language model (LLM) to generate text in a target language from textual prompts in the first language;
• processing, using the LLM, the textual prompt conditioned on the fine-tuned prompt embedding to generate output text in the target language;
• concatenating the textual prompt and the generated output text to provide an unspoken textual utterance;
• and
• training a multilingual automatic speech recognition (ASR) model to learn how to recognize speech in the target language by injecting the unspoken textual utterance into a text encoder associated with the multilingual ASR model.