# US 12,573,401 B1 · Neural network based conversation-aware automatic speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,573,401 B1
- Document ID: US12573401B1
- Status: Granted
- Kind: B1
- Application number: 17/246,250
- Filed: 2021-04-30
- Granted: 2026-03-10
- Earliest claimed: 2021-04-30
- Assignees: Interactions LLC
- Applicants: Interactions LLC
- Inventors: Shahab Jalalvand
- Web page: https://patentgrep.com/patent/US12573401B1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06N 3/08: Learning methods
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/063: {Training}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 25/30: using neural networks
- G10L 15/16: using artificial neural networks
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system uses a machine learning based model such as a neural network for transcribing audio inputs. The system receives a set of audio inputs representing utterances of a conversation. For each conversation, the system determines a dialogue state for each utterance. The system uses a hierarchical language model for transcribing audio inputs of an online conversation using the received conversations. The hierarchical language model includes a top-level language model and a plurality of lower-level language model. The training is performed by (1) training the top-level language model using sequences of corresponding dialogue state, each sequence of dialogue states for a conversation, and (2) for each dialogue state, training a lower-level language model using utterances having that dialogue state. The system executes the hierarchical language model to transcribe audio input of new conversations.

## Claims

### Claim 1

A computer-implemented method for generating a machine learning based model for transcribing audio inputs, the computer-implemented method comprising:
receiving a set of audio inputs;
determining a set of conversations from the set of audio inputs, wherein a conversation comprises a plurality of rounds, each round representing an exchange of utterances between participants of the conversation;
for each conversation, determining a respective dialogue state for each utterance of the conversation, wherein each respective dialogue state corresponds to a set of rounds of the conversation that are related, the dialogue state corresponding to a cluster of utterance embeddings obtained by encoding utterances using a hierarchical encoding model comprising
(1) a plurality of low-level encoding models, each low-level encoding model configured to generate a context-independent embedding for a dialog round, and
(2) a high-level encoding model configured to receive as input a set of context-independent embeddings for a dialog round and output a contextual embedding for the dialog round, wherein training the low-level encoding model and the high-level encoding model includes using a set of positive and negative examples, wherein a positive example includes a dialog round and a positive response generated based on an actual response to the dialog round, and a negative example includes the dialog round and a negative response generated based on a random response from a corpus of dialogs;
training a hierarchical language model for transcribing audio inputs of a conversation using the set of conversations, the hierarchical language model comprising a top-level language model and a plurality of lower-level language models, the training comprising:
training the top-level language model using sequences of the respective dialogue states determined for the set of conversations, each sequence of dialogue states for a particular conversation in the set of conversations, the top-level language model trained to predict a probability of occurrence of a particular dialogue state, given a sequence of previous dialogue states in a corresponding conversation, and
for each dialogue state, training a plurality of lower-level language models, wherein each lower language model is trained for a source of utterance, each lower language model trained using utterances from the source having that dialogue state, the lower-level language model trained to predict the probability of occurrence of a word based on an input utterance;
receiving a new audio input for a new conversation; and
executing the hierarchical language model to transcribe the new audio input, the executing comprising, predicting the particular dialogue state using the top-level language model, and using a lower-level language model trained for a source of utterance, the lower-level language model corresponding to the predicted dialogue state to predict a word based on an utterance of the new audio input.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein each conversation is between a particular agent and a particular user, and wherein determining the particular dialogue state for an utterance of the conversation comprises:
segmenting the conversation into a set of rounds, wherein each round represents a pair of consecutive utterances comprising an utterance by the particular agent and an utterance by the particular user;
performing clustering of the set of rounds to determine a plurality of clusters; and
associating each cluster in the plurality of clusters with the particular dialogue state.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein performing clustering of the set of rounds comprises:
determining a set of embeddings, wherein an embedding is for a round; and
performing clustering of the set of embeddings.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, further comprising:
identifying each dialogue state with a set of keywords based on a subset of utterances having the dialogue state.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising: determining the set of keywords for the particular dialogue state based on term frequency-inverse document frequency of keywords occurring in utterances having the particular dialogue state.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the particular dialogue state is associated with a particular cluster in the plurality of clusters, wherein determining the set of keywords for the particular dialogue state comprises:
receiving a set of conversations;
for each conversation in the set of conversations, for each round in the conversation, splitting the round into an agent utterance and a user utterance;
extracting a first set of keywords from agent utterances belonging to the particular cluster;
extracting a second set of keywords from user utterances belonging to the particular cluster; and
combining the first set of keywords and the second set of keywords as the set of keywords.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the top-level language model is an n-gram model trained on sequences of dialogue states.

### Claim 8

A non-transitory computer-readable storage medium storing a hierarchical language model comprising a top-level language model and a plurality of lower-level language models, wherein the hierarchical language model is trained using steps comprising:
receiving a set of audio inputs;
determining a set of conversations from the set of audio inputs, wherein a conversation comprises a plurality of rounds, each round representing an exchange of utterances between participants of the conversation;
for each conversation, determining a respective dialogue state for each utterance of the conversation, wherein each respective dialogue state corresponds to a set of rounds of the conversation that are related, the dialogue state corresponding to a cluster of utterance embeddings obtained by encoding utterances using a hierarchical encoding model comprising
(1) a plurality of low-level encoding models, each low-level encoding model configured to generate a context-independent embedding for a dialog round, and
(2) a high-level encoding model configured to receive as input a set of context-independent embeddings for a dialog round and output a contextual embedding for the dialog round, wherein training the low-level encoding model and the high-level encoding model includes using a set of positive and negative examples, wherein a positive example includes a dialog round and a positive response generated based on an actual response to the dialog round, and a negative example includes the dialog round and a negative response generated based on a random response from a corpus of dialogs;
training a hierarchical language model for transcribing audio inputs of a conversation using the set of conversations, the hierarchical language model comprising a top-level language model and a plurality of lower-level language models, the training comprising:
training the top-level language model using sequences of the respective dialogue states determined for the set of conversations, each sequence of dialogue states for a particular conversation in the set of conversations, the top-level language model trained to predict a probability of occurrence of a particular dialogue state, given a sequence of previous dialogue states in a corresponding conversation, and
for each dialogue state, training a plurality of lower-level language models, wherein each lower language model is trained for a source of utterance, each lower language model trained using utterances from the source having that dialogue state, the lower-level language model trained to predict the probability of occurrence of a word based on an input utterance;
receiving a new audio input for a new conversation; and
executing the hierarchical language model to transcribe the new audio input, the executing comprising, predicting the particular dialogue state using the top-level language model, and using a lower-level language model trained for a source of utterance, the lower-level language model corresponding to the predicted dialogue state to predict a word based on an utterance of the new audio input.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein each conversation is between a particular agent and a particular user, and wherein determining a corresponding dialogue state for each utterance of the conversation comprises:
segmenting the conversation into a set of rounds, wherein each round represents a pair of consecutive utterances comprising an utterance by the particular agent and an utterance by the particular user;
performing clustering of the set of rounds to determine a plurality of clusters; and
associating each cluster in the plurality of clusters with the corresponding dialogue state.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein performing clustering of the set of rounds comprises:
determining a set of embeddings, wherein an embedding is for a round; and
performing clustering of the set of embeddings.

### Claim 11 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the hierarchical language model is trained using steps further comprising:
identifying each dialogue state with a set of keywords based on a subset of utterances having the dialogue state.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the hierarchical language model is trained using steps further comprising:
determining the set of keywords for a particular dialogue state based on term frequency-inverse document frequency of keywords occurring in utterances having the dialogue state.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the particular dialogue state is associated with a particular cluster in the plurality of clusters, wherein determining the set of keywords for the particular dialogue state comprises:
receiving a set of conversations;
for each conversation in the set of conversations, for each round in the conversation, splitting the round into an agent utterance and a user utterance;
extracting a first set of keywords from agent utterances belonging to the particular cluster;
extracting a second set of keywords from user utterances belonging to the particular cluster; and
combining the first set of keywords and the second set of keywords as the set of keywords.

### Claim 14 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the top-level language model is an n-gram model trained on sequences of dialogue states.

### Claim 15

A computer system comprising:
a computer processor; and
a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor cause the computer processor to perform steps comprising:
receiving a set of audio inputs;
receiving a set of audio inputs;
determining a set of conversations from the set of audio inputs, wherein a conversation comprises a plurality of rounds, each round representing an exchange of utterances between participants of the conversation;
for each conversation, determining a respective dialogue state for each utterance of the conversation, wherein each respective dialogue state corresponds to a set of rounds of the conversation that are related, the dialogue state corresponding to a cluster of utterance embeddings obtained by encoding utterances using a hierarchical encoding model comprising
(1) a plurality of low-level encoding models, each low-level encoding model configured to generate a context-independent embedding for a dialog round, and
(2) a high-level encoding model configured to receive as input a set of context-independent embeddings for a dialog round and output a contextual embedding for the dialog round, wherein training the low-level encoding model and the high-level encoding model includes using a set of positive and negative examples, wherein a positive example includes a dialog round and a positive response generated based on an actual response to the dialog round, and a negative example includes the dialog round and a negative response generated based on a random response from a corpus of dialogs;
training a hierarchical language model for transcribing audio inputs of a conversation using the set of conversations, the hierarchical language model comprising a top-level language model and a plurality of lower-level language models, the training comprising:
training the top-level language model using sequences of the respective dialogue states determined for the set of conversations, each sequence of dialogue states for a particular conversation in the set of conversations, the top-level language model trained to predict a probability of occurrence of a particular dialogue state, given a sequence of previous dialogue states in a corresponding conversation, and
for each dialogue state, training a plurality of lower-level language models, wherein each lower language model is trained for a source of utterance, each lower language model trained using utterances from the source having that dialogue state, the lower-level language model trained to predict the probability of occurrence of a word based on an input utterance;
receiving a new audio input for a new conversation; and
executing the hierarchical language model to transcribe the new audio input, the executing comprising, predicting the particular dialogue state using the top-level language model, and using a lower-level language model trained for a source of utterance, the lower-level language model corresponding to the predicted dialogue state to predict a word based on an utterance of the new audio input.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein each conversation is between a particular agent and a particular user, and wherein determining the particular dialogue state for an utterance of the conversation comprises:
segmenting the conversation into a set of rounds, wherein each round represents a pair of consecutive utterances comprising an utterance by the particular agent and an utterance by the particular user;
performing clustering of the set of rounds to determine a plurality of clusters; and
associating each cluster in the plurality of clusters with the particular dialogue state.

### Claim 17 (depends on 16)

The computer system of claim 16, wherein performing clustering of the set of rounds comprises:
determining a set of embeddings, wherein an embedding is for a round; and
performing clustering of the set of embeddings.

### Claim 18 (depends on 16)

The computer system of claim 16, wherein the instructions further cause the computer processor to perform steps comprising:
identifying each dialogue state with a set of keywords based on a subset of utterances having the dialogue state.

### Claim 19 (depends on 18)

The computer system of claim 18, wherein the instructions further cause the computer processor to perform steps comprising:
determining the set of keywords for the particular dialogue state based on term frequency-inverse document frequency of keywords occurring in utterances having the particular dialogue state.

### Claim 20 (depends on 19)

The computer system of claim 19, wherein the particular dialogue state is associated with a particular cluster in the plurality of clusters, wherein determining the set of keywords for the particular dialogue state further comprises:
receiving a set of conversations;
for each conversation in the set of conversations, for each round in the conversation, splitting the round into an agent utterance and a user utterance;
extracting a first set of keywords from agent utterances belonging to the particular cluster;
extracting a second set of keywords from user utterances belonging to the particular cluster; and
combining the first set of keywords and the second set of keywords as the set of keywords.
