# US 2019/0318731 A1 · Collaborative Transcription With Bidirectional Automatic Speech Recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318731 A1
- Document ID: US20190318731A1
- Status: Published application
- Kind: A1
- Application number: 15/951,719
- Filed: 2018-04-12
- Published: 2019-10-17
- Earliest claimed: 2018-04-12
- Assignees: Nuance Communications, Inc.
- Applicants: Nuance Communications, Inc.
- Inventors: Uwe Helmut Jost; Neeraj Deshmukh
- Web page: https://patentgrep.com/patent/US20190318731A1

## CPC

- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 15/05: Word boundary detection
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method of performing bidirectional automatic speech recognition (ASR) using an external information source includes performing a precompute pass by pre-processing an utterance in a backward direction to generate pre-processing data stored in a data structure. In a run-time pass, ASR is performed on the utterance in a forward direction using the pre-processing data to generate a prediction list that has a given number of words in path probability order. A word prediction based on the prediction list is presented to an external information source to obtain a response confirming, selecting or correcting the word prediction. The word prediction based on the response and the prediction list are updated. Processing repeats until the end of the utterance is reached. The method outputs an automatic speech recognized form of the utterance based on the word prediction. Use of the external information source in an integrated manner improves current and future predictions.

## Claims

### Claim 1

A method of performing bidirectional automatic speech recognition using an external information source, the method comprising:
performing a precompute pass by:
a) pre-processing an utterance in a backward direction from end to start of the utterance to generate pre-processing data stored in a data structure;
performing a run-time pass by:
b) performing automatic speech recognition on the utterance in a forward direction using the pre-processing data to generate a prediction list that has a given number of words in path probability order;
c) (i) presenting a word prediction based on the prediction list to an external information source to obtain a response from the external information source confirming, selecting or correcting the word prediction, (ii) updating the word prediction based on the response from the external information source, and (iii) updating the prediction list accordingly; and
d) repeating b) and c) until the end of the utterance is reached; and
outputting an automatic speech recognized form of the utterance based on the word prediction.

### Claim 2 (depends on 1)

The method of claim 1, wherein the external information source is a human agent.

### Claim 3 (depends on 1)

The method of claim 1, wherein the word prediction includes n-best possible words.

### Claim 4 (depends on 1)

The method of claim 1, further comprising updating a model employed by the automatic speech recognition in the forward direction as a result of the response from the external information source.

### Claim 5 (depends on 4)

The method of claim 4, wherein updating the model includes one or more of adding a word to a vocabulary, updating a lexical cache model, incrementally building a document specific language model from recognized text for interpolation with an original language model, or adapting acoustic model parameters using information gained by aligning a new word with audio data of the utterance.

### Claim 6 (depends on 1)

The method of claim 1, wherein pre-processing the utterance in a backward direction includes performing automatic speech recognition on the utterance with a reverse language model.

### Claim 7 (depends on 1)

The method of claim 1, wherein the utterance is divided into frames, and wherein the data structure includes, for each frame of the utterance, a path score of a best path to the end of the utterance from the frame.

### Claim 8 (depends on 7)

The method of claim 7, wherein the data structure includes, for each word that ends in a given frame, (1) a combined score of acoustic model and language model scores for the best path to the end of the utterance from the frame, and (2) a minimum score of the combined acoustic model and language model scores over all words that end in the frame.

### Claim 9 (depends on 8)

The method of claim 8, wherein the data structure further includes acoustic parameters.

### Claim 10 (depends on 1)

The method of claim 1, wherein, after updating the word prediction, the automatic speech recognition in the forward direction is performed from a starting point earlier in time than a word start of a word just predicted or confirmed and is initially restricted to a sequence including at least the just predicted or corrected word or more words that have already been confirmed.

### Claim 11 (depends on 10)

The method of claim 10, wherein the starting point is selected based on the start time of the first confirmed word in the sequence.

### Claim 12 (depends on 10)

The method of claim 10, wherein the automatic speech recognition in the forward direction is performed until one or more ends of new words are hypothesized by the automatic speech recognition.

### Claim 13 (depends on 12)

The method of claim 12, further comprising looking up the hypothesized word ends in the data structure to determine, for each of the word ends, whether the word end is found at a given frame in the data structure; and
(i) if the word end is found at the frame, reading relevant scores from the data structure and combining them with the current forward scores to calculate an overall score for the whole utterance;
(ii) if the word is not found at the frame, assigning a value not higher than the minimum score as the overall score.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
pausing automatic speech recognition in the forward direction when any remaining active hypotheses have scores below a predetermined threshold or when a timeout is reached; and
presenting the top n hypotheses according to the overall scores for the whole utterance to the external information source for confirmation, selection or correction.

### Claim 15 (depends on 1)

The method of claim 1, wherein performing the automatic speech recognition in the forward direction includes linking a forward search space with a subset of the pre-processing data.

### Claim 16

A system for performing bidirectional automatic speech recognition using an external information source, the system comprising:
a memory storing computer code instructions thereon; and
a processor,
the memory, with the computer code instructions, and the processor being configured to cause the system to:
perform a precompute pass by:
a) pre-processing an utterance in a backward direction from end to start of the utterance to generate pre-processing data stored in a data structure;
perform a run-time pass by:
b) performing automatic speech recognition on the utterance in a forward direction using the pre-processing data to generate a prediction list that has a given number of words in path probability order;
c) (i) presenting a word prediction based on the prediction list to an external information source to obtain a response from the external information source confirming, selecting or correcting the word prediction, (ii) updating the word prediction based on the response from the external information source, and (iii) updating the prediction list accordingly; and
d) repeating b) and c) until the end of the utterance is reached; and
output an automatic speech recognized form of the utterance based on the word prediction.

### Claim 17 (depends on 16)

The system of claim 16, wherein the memory, with computer code instructions, and the processor are configured further to update a model employed by the automatic speech recognition in the forward direction as a result of the response from the external information source.

### Claim 18 (depends on 16)

The system of claim 16, wherein the system comprises a server and an agent device in communication with the server, and wherein the memory comprises a server memory and an agent memory, and wherein the processor comprises a server processor and an agent processor, and wherein the server memory, with the computer code instructions, and the server processor are configured to cause the server to perform the precompute pass, and wherein the agent memory, with the computer code instructions, and the agent processor are configured to cause the agent device to perform the run-time pass and to output the automatic speech recognized form of the utterance.

### Claim 19

A non-transitory computer-readable medium including computer code instructions stored thereon for performing bidirectional automatic speech recognition using an external information source, the computer code instructions, when executed by a processor, cause a system to perform at least the following:
perform a precompute pass by:
a) pre-processing an utterance in a backward direction from end to start of the utterance to generate pre-processing data stored in a data structure;
perform a run-time pass by:
b) performing automatic speech recognition on the utterance in a forward direction using the pre-processing data to generate a prediction list that has a given number of words in path probability order;
c) (i) presenting a word prediction based on the prediction list to an external information source to obtain a response from the external information source confirming, selecting or correcting the word prediction, (ii) updating the word prediction based on the response from the external information source, and (iii) updating the prediction list accordingly; and
d) repeating b) and c) until the end of the utterance is reached; and
output an automatic speech recognized form of the utterance based on the word prediction.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the computer code instructions, when executed by the processor, cause the system further to update a model employed by the automatic speech recognition in the forward direction as a result of the response from the external information source.
