# US 2023/0056680 A1 · Integrating dialog history into end-to-end spoken language understanding systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056680 A1
- Document ID: US20230056680A1
- Status: Published application
- Kind: A1
- Application number: 17/405,532
- Filed: 2021-08-18
- Published: 2023-02-23
- Earliest claimed: 2021-08-18
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Samuel Thomas; Jatin Ganhotra; Hong-Kwang Kuo; Sachindra Joshi; George Andrei Saon; Zoltan Tueske; Brian E. D. Kingsbury
- Web page: https://patentgrep.com/patent/US20230056680A1

## CPC

- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/065: Adaptation
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 2015/088: {Word spotting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Audio signals representing a current utterance in a conversation and a dialog history including at least information associated with past utterances corresponding to the current utterance in the conversation can be received. The dialog history can be encoded into an embedding. A spoken language understanding neural network model can be trained to perform a spoken language understanding task based on input features including at least speech features associated with the received audio signals and the embedding. An encoder can also be trained to encode a given dialog history into an embedding. The spoken language understanding task can include predicting a dialog action of an utterance. The spoken language understanding task can include predicting a dialog intent or overall topic of the conversation.

## Claims

### Claim 1

A system comprising:
at least one processor; and
at least one memory device coupled with the processor;
the at least one processor configured to:
receive audio signals representing a current utterance in a conversation and a dialog history including at least information associated with past utterances corresponding to the current utterance in the conversation;
encode the dialog history into an embedding; and
train a spoken language understanding neural network model to perform a spoken language understanding task based on input features including at least speech features associated with the received audio signals and the embedding.

### Claim 2 (depends on 1)

The system of claim 1, wherein the dialog history includes text, said at least one processor configured to encode the text into the embedding using a text encoder.

### Claim 3 (depends on 2)

The system of claim 2, wherein the encoder is trained on application-specific data and task.

### Claim 4 (depends on 1)

The system of claim 1, wherein the dialog history includes audio signals, wherein said at least one processor is configured to encode the dialog history into the embedding directly from the audio signals.

### Claim 5 (depends on 1)

The system of claim 1, wherein at least some of the dialog history includes machine inferred information associated with the past utterances.

### Claim 6 (depends on 1)

The system of claim 1, wherein at least some of the dialog history includes output of the spoken language understanding neural network model from a previous turn in the conversation.

### Claim 7 (depends on 1)

The system of claim 1, wherein the embedding includes at least information representing any one or combinations of speaker roles associated with the dialog history, dialog actions associated with the past utterances in the dialog history, sentiment associated with the past utterances in the dialog history, automated action performed by the at least one processor associated with processing the dialog history, and transcription associated with the past utterances.

### Claim 8 (depends on 1)

The system of claim 1, wherein said at least one processor is configured to train the spoken language understanding neural network model based on a plurality of audio signals representing current utterances, a plurality of sets of past utterances representing dialog histories associated with the current utterances in the conversation, and a plurality of encoded embeddings associated with the dialog histories.

### Claim 9 (depends on 1)

The system of claim 1, wherein the spoken language understanding neural network model includes end-to-end neural network accommodating spoken language understanding labels.

### Claim 10 (depends on 1)

The system of claim 1, wherein the spoken language understanding neural network model integrating the dialog history is part of, and customizes, a speech-based spoken language understanding tool.

### Claim 11 (depends on 1)

The system of claim 1, wherein the spoken language understanding task includes predicting a dialog action associated with the current utterance.

### Claim 12 (depends on 1)

The system of claim 1, wherein the spoken language understanding task includes predicting a dialog intent associated with the conversation.

### Claim 13 (depends on 11)

The system of claim 11, wherein said at least one processor is further configured to autonomously generate a next conversation utterance based on the spoken language understanding task output by the trained spoken language understanding neural network model.

### Claim 14 (depends on 13)

The system of claim 13, wherein said at least one processor is further configured to output as speech the generated next conversation utterance in conversing with a user.

### Claim 15

A method for spoken language understanding, comprising:
receiving audio signals representing a current utterance in a conversation and a dialog history including at least information associated with past utterances corresponding to the current utterance in the conversation;
encoding the dialog history into an embedding; and
training a spoken language understanding neural network model to perform a spoken language understanding task based on input features including at least speech features associated with the received audio signals and the embedding.

### Claim 16 (depends on 15)

The method of claim 15, wherein the dialog history includes text, and the encoding includes encoding the text into the embedding using a text encoder.

### Claim 17 (depends on 16)

The method of claim 16, further including training the encoder on application-specific data and task to encode the dialog history into the embedding.

### Claim 18 (depends on 15)

The method of claim 15, wherein at least some of the dialog history includes machine inferred information associated with the past utterances.

### Claim 19 (depends on 15)

The method of claim 15, wherein at least some of the dialog history includes output of the spoken language understanding neural network model from a previous utterance turn.

### Claim 20

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a device to cause the device to:
receive audio signals representing a current utterance in a conversation and a dialog history including at least information associated with past utterances corresponding to the current utterance in the conversation;
encode the dialog history into an embedding; and
train a spoken language understanding neural network model to perform a spoken language understanding task based on input features comprising speech features associated with the received audio signals and the embedding.
