Enabling large language model-based spoken language understanding (slu) systems to leverage both audio data and textual data in processing spoken utterances

A method implemented by one or more processors of a computing device, the method comprising:

• obtaining a training example, wherein the training example comprises audio data that captures a spoken utterance and a label indicative of a semantic representation corresponding to the spoken utterance;
• processing the audio data using an automatic speech recognition model to generate textual data corresponding to the spoken utterance;
• fine-tuning a pre-trained large language model (LLM) to generate a fine-tuned LLM based on applying both the audio data and the textual data as input across the LLM, the pre-trained LLM comprising (i) a pre-trained audio encoder that is pre-trained to generate audio embeddings representing audio data and (ii) a pre-trained text encoder that is pre-trained to generate textual embeddings representing textual data;
• and
• subsequent to generating the fine-tuned LLM, causing the fine-tuned LLM to be deployed in processing additional spoken utterances.

A system comprising:

• at least one processor;
• and
• memory storing instructions that, when executed, cause the at least one processor to perform operations, the operations comprising:
• receiving audio data that captures a spoken utterance of a user;
• processing the audio data using an automatic speech recognition (ASR) model to generate textual data corresponding to the spoken utterance;
• generating a semantic representation corresponding to the spoken utterance of the user based on applying both the audio data and the textual data as input across a large language model (LLM);
• and
• causing the semantic representation corresponding to the spoken utterance of the user to be utilized in fulfilling the spoken utterance.