# US 2024/0203404 A1 · Enabling large language model-based spoken language understanding (slu) systems to leverage both audio data and textual data in processing spoken utterances

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203404 A1
- Document ID: US20240203404A1
- Status: Published application
- Kind: A1
- Application number: 18/081,569
- Filed: 2022-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Nir Shabat; Volodymyr Polosukhin; Shlomo Fruchter; Golan Pundak; Roy Atsmon
- Web page: https://patentgrep.com/patent/US20240203404A1

## CPC

- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/183: using context dependencies, e.g. language models
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/30: Semantic analysis
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/096: Transfer learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G10L 15/16: using artificial neural networks
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G10L 2015/228: {of application context}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In various implementations, a method implemented by one or more processors of a computing device can comprise receiving audio data that captures a spoken utterance of a user; processing the audio data using an automatic speech recognition (ASR) model to generate textual data corresponding to the spoken utterance; generating a semantic representation corresponding to the spoken utterance of the user based on applying both the audio data and the textual data as input across a large language model (LLM); and causing the semantic representation corresponding to the spoken utterance of the user to be utilized in fulfilling the spoken utterance.

## Claims

### Claim 1

A method implemented by one or more processors of a computing device, the method comprising:
receiving audio data that captures a spoken utterance of a user;
processing the audio data using an automatic speech recognition (ASR) model to generate textual data corresponding to the spoken utterance;
generating a semantic representation corresponding to the spoken utterance of the user based on applying both the audio data and the textual data as input across a large language model (LLM); and
causing the semantic representation corresponding to the spoken utterance of the user to be utilized in fulfilling the spoken utterance.

### Claim 2 (depends on 1)

The method of claim 1, wherein the semantic representation corresponding to the spoken utterance comprises a user intent.

### Claim 3 (depends on 1)

The method of claim 1, wherein applying both the audio data and the textual data as input across the large language model comprises:
determining one or more audio encodings representing the audio data;
determining one or more textual encodings representing the textual data;
combining the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data.

### Claim 4 (depends on 3)

The method of claim 3, wherein combining the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data comprises at least one of:
concatenating the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data, and
aggregating the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data by:
using an attention mechanism to generate a context vector from the textual data to the audio data; and
summing the context vector with the one or more textual encodings representing the textual data.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
for each of a plurality of frames in the audio data, determining a corresponding encoding representing the frame in the audio data; and
aggregating the corresponding encodings representing each of the frames in the audio data to determine an aggregated audio encoding of a fixed dimension representing the audio data.

### Claim 6 (depends on 5)

The method of claim 5 wherein combining the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data comprises:
summing the aggregated audio encoding representing the audio data and the textual encoding representing the textual data.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating refined textual content corresponding to the spoken utterance based at least in part on the semantic representation corresponding to the spoken utterance.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
causing the refined textual content to be utilized in fulfilling the spoken utterance.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
generating synthesized speech of a reply to the spoken utterance based on the semantic representation corresponding to the spoken utterance of the user; and
causing the synthesized speech to be provided for presentation to the user.

### Claim 10 (depends on 9)

The method of claim 9, wherein causing the synthesized speech to be provided for presentation to the user comprises:
causing the synthesized speech to be provided for audible presentation to the user.

### Claim 11 (depends on 1)

The method of claim 1, wherein the spoken utterance from the user includes a request to perform a task, and fulfilling the spoken utterance comprises:
causing one or more actions to be performed in furtherance of completing the task.

### Claim 12 (depends on 1)

The method of claim 1, wherein the LLM comprises a text encoder and an audio encoder.

### Claim 13 (depends on 1)

The method of claim 1, wherein the LLM has been fine-tuned using domain specific training data for a domain, and the spoken utterance relates to the domain.

### Claim 14 (depends on 13)

The method of claim 13, wherein the domain relates to causing performance of one or more tasks via a telephone conversation, and the LLM has been fine-tuned for automating telephone conversations for causing performance of the one or more tasks.

### Claim 15

A method implemented by one or more processors of a computing device, the method comprising:
obtaining a training example, wherein the training example comprises audio data that captures a spoken utterance and a label indicative of a semantic representation corresponding to the spoken utterance;
processing the audio data using an automatic speech recognition model to generate textual data corresponding to the spoken utterance;
fine-tuning a pre-trained large language model (LLM) to generate a fine-tuned LLM based on applying both the audio data and the textual data as input across the LLM, the pre-trained LLM comprising (i) a pre-trained audio encoder that is pre-trained to generate audio embeddings representing audio data and (ii) a pre-trained text encoder that is pre-trained to generate textual embeddings representing textual data; and
subsequent to generating the fine-tuned LLM, causing the fine-tuned LLM to be deployed in processing additional spoken utterances.

### Claim 16 (depends on 15)

The method of claim 15, wherein the audio data in the training example is part of a previous telephone conversation related to a particular domain.

### Claim 17

A system comprising:
at least one processor; and
memory storing instructions that, when executed, cause the at least one processor to perform operations, the operations comprising:
receiving audio data that captures a spoken utterance of a user;
processing the audio data using an automatic speech recognition (ASR) model to generate textual data corresponding to the spoken utterance;
generating a semantic representation corresponding to the spoken utterance of the user based on applying both the audio data and the textual data as input across a large language model (LLM); and
causing the semantic representation corresponding to the spoken utterance of the user to be utilized in fulfilling the spoken utterance.

### Claim 18 (depends on 17)

The system of claim 17, wherein the semantic representation corresponding to the spoken utterance comprises a user intent.

### Claim 19 (depends on 17)

The system of claim 17, wherein applying both the audio data and the textual data as input across the large language model comprises:
determining one or more audio encodings representing the audio data;
determining one or more textual encodings representing the textual data;
combining the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data.

### Claim 20 (depends on 19)

The system of claim 19, wherein combining the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data comprises at least one of:
concatenating the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data, and
aggregating the one or more audio encodings representing the audio data and the one or more textual encodings representing the textual data by:
using an attention mechanism to generate a context vector from the textual data to the audio data; and
summing the context vector with the one or more textual encodings representing the textual data.
