# US 2023/0402034 A1 · Correcting speech recognition errors by considering prior user edits and/or assessing fulfillment data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0402034 A1
- Document ID: US20230402034A1
- Status: Published application
- Kind: A1
- Application number: 17/842,166
- Filed: 2022-06-16
- Published: 2023-12-14
- Earliest claimed: 2022-06-13
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ágoston Weisz; Miroslaw Michalski; Aurélien Boffy
- Web page: https://patentgrep.com/patent/US20230402034A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/063: {Training}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Implementations relate to correcting a speech recognition hypothesis based on prior correction(s) made by a user and/or fulfillment data associated with fulfilling a request embodied in the speech recognition hypothesis. A candidate speech recognition hypothesis can be generated in response to the user providing a spoken utterance to an application, such as an automated assistant. When a confidence metric for the candidate speech recognition hypothesis does not satisfy a threshold, one or more terms of the candidate speech recognition hypothesis can be compared to correcting data. The correcting data can indicate whether the user previously corrected any term(s) present in the candidate speech recognition hypothesis and, if so, correct the term(s) accordingly. Fulfillment data generated for the candidate hypothesis and/or for the corrected hypothesis can also be processed to determine whether to utilize the candidate hypothesis or the corrected hypothesis in responding to the user.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
processing audio data that captures a spoken utterance provided by a user to an automated assistant that is accessible via a computing device,
wherein the spoken utterance embodies a request to cause the automated assistant to perform one or more operations in furtherance of fulfilling the request;
generating, based on processing the audio data, speech recognition hypothesis data for the spoken utterance,
wherein the speech recognition hypothesis data characterizes a speech recognition hypothesis and an alternate speech recognition hypothesis that are each a corresponding prediction of the spoken utterance,
wherein the alternate speech recognition hypothesis includes one or more correcting terms that are not included in the speech recognition hypothesis, the speech recognition hypothesis includes one or more corrected terms that are not in the alternate speech recognition hypothesis, and
wherein the one or more correcting terms are included, in the alternate speech recognition hypothesis in lieu of the corrected terms, based on one or more prior instance of correction of the one or more corrected terms to the one or more correcting terms;
determining, based on fulfillment data, whether to cause the automated assistant to respond to the speech recognition hypothesis or to instead respond to the alternate speech recognition hypothesis,
wherein the fulfillment data indicates responsiveness of the automated assistant to the speech recognition hypothesis and/or to the alternate speech recognition hypothesis; and
causing, based on the determining, the automated assistant to respond to either the speech recognition hypothesis or the alternate speech recognition hypothesis.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the fulfillment data includes hypothesis fulfillment data corresponding to the speech recognition hypothesis and includes alternate fulfillment data corresponding to the alternate speech recognition hypothesis, and
wherein determining whether to cause the automated assistant to respond to the speech recognition hypothesis or to instead respond to the alternate speech recognition hypothesis is based on the hypothesis fulfillment data and is based on the alternate fulfillment data.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining whether to cause the automated assistant to respond to the speech recognition hypothesis or to instead respond to the alternate speech recognition hypothesis includes:
comparing the hypothesis fulfillment data to the alternate fulfillment data.

### Claim 4 (depends on 3)

The method of claim 3, wherein the hypothesis fulfillment data indicates a first non-binary degree to which the automated assistant is predicted to be responsive to the speech recognition hypothesis, wherein the alternate fulfillment data indicates a second non-binary degree to which the automated assistant is predicted to be responsive to the alternate speech recognition hypothesis, and wherein comparing the hypothesis fulfillment data to the alternate fulfillment data includes:
comparing the first non-binary degree to the second non-binary degree, while biasing toward determining to cause the automated assistant to respond to the speech recognition hypothesis in response to the speech recognition hypothesis being a primary hypothesis generated based on processing the audio data.

### Claim 5 (depends on 2)

The method of claim 2, wherein determining whether to cause the automated assistant to respond to the speech recognition hypothesis or to instead respond to the alternate speech recognition hypothesis includes:
generating first output based on processing at least the hypothesis fulfillment data using a machine learning model;
generating second output based on processing at least the alternate fulfillment data using the machine learning model; and
determining, based on comparing the first output to the second output, whether to cause the automated assistant to respond to the speech recognition hypothesis or to instead respond to the alternate speech recognition hypothesis.

### Claim 6 (depends on 5)

The method of claim 5, wherein comparing the first output to the second output comprises biasing toward the first output in response to the first output being for the speech recognition hypothesis and the speech recognition hypothesis being a primary hypothesis generated based on processing the audio data.

### Claim 7 (depends on 5)

The method of claim 5, wherein generating the first output is further based on processing, using the machine learning model and along with the hypothesis fulfillment data, current contextual data, and wherein generating the second output is further based on processing, using the machine learning model and along with the alternate fulfillment data, the current contextual data.

### Claim 8 (depends on 7)

The method of claim 7, wherein the current contextual data includes a current location of the computing device and/or current temporal data.

### Claim 9 (depends on 7)

The method of claim 7, wherein generating the first output is further based on processing, using the machine learning model and along with the hypothesis fulfillment data, one or more automatic speech recognition metrics generated during processing the audio data to generate the speech recognition hypothesis.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating the speech recognition hypothesis data comprises:
processing the audio data using one or more automatic speech recognition models;
determining the speech recognition hypothesis based on it being a hypothesis generated based on processing the audio data using the one or more automatic speech recognition models;
determining, based on correction data associated with the computing device and/or the user, that the one or more corrected terms, of the speech recognition hypothesis, were previously corrected, during one or more prior interactions at the computing device and/or by the user, to the one or more correcting terms of the alternate speech recognition hypothesis; and
based on determining that the one or more corrected terms were previously corrected to the one or more correcting terms:
generating the alternate speech recognition hypothesis by replacing the one or more corrected terms with the one or more correcting terms.

### Claim 11 (depends on 10)

The method of claim 10, wherein the one or more prior interactions include a prior interaction in which the one or more correcting terms were manually entered, using a keyboard interface, and were manually entered as a replacement to the one or more corrected terms after display of the one or more corrected terms.

### Claim 12 (depends on 10)

The method of claim 10, wherein the fulfillment data includes hypothesis fulfillment data that indicates responsiveness of the automated assistant to the speech recognition hypothesis, and wherein generating the alternate speech recognition hypothesis is further based on determining that the hypothesis fulfillment data fails to satisfy one or more criteria.

### Claim 13 (depends on 10)

The method of claim 10, wherein determining, based on data associated with the computing device and/or the user, that the one or more corrected terms were previously corrected, during one or more prior interactions at the computing device and/or by the user, to the one or more correcting terms of the alternate speech recognition hypothesis, comprises:
determining that current contextual data, when processing the audio data, matches prior contextual data of the one or more prior interactions; and
wherein generating the alternate speech recognition hypothesis is further based on determining that the current contextual data matches the prior contextual data.

### Claim 14

A method implemented by one or more processors, the method comprising:
causing a speech recognition hypothesis to be processed using one or more trained machine learning models,
wherein the speech recognition hypothesis is a prediction of a spoken utterance provided by a user to an automated assistant that is accessible via a computing device;
determining, based on the speech recognition hypothesis being processed using the one or more trained machine learning models, that a hypothesis metric for the speech recognition hypothesis satisfies a misrecognition threshold;
determining, based on the hypothesis metric satisfying the misrecognition threshold, a separate speech recognition hypothesis,
wherein the separate speech recognition hypothesis includes one or more correcting terms that are not provided in the speech recognition hypothesis, and the speech recognition hypothesis includes one or more hypothesis terms that are not provided in the separate speech recognition hypothesis;
causing the separate speech recognition hypothesis to be processed using the one or more trained machine learning models,
wherein the separate speech recognition hypothesis is a separate prediction of the spoken utterance provided by the user to the automated assistant; and
determining, based on the separate speech recognition hypothesis being processed using the one or more trained machine learning models, that a separate hypothesis metric for the separate speech recognition hypothesis does not satisfy the misrecognition threshold or does not satisfy an alternate threshold; and
causing, based on the separate hypothesis metric satisfying the misrecognition threshold, the automated assistant to initialize performance of one or more operations in furtherance of satisfying a request embodied in the separate speech recognition hypothesis.

### Claim 15 (depends on 14)

The method of claim 14, wherein the separate speech recognition hypothesis is further determined based on the one or more correcting terms being previously provided by the user to the automated assistant during a previous interaction between the user and the automated assistant.

### Claim 16 (depends on 15)

The method of claim 15,
wherein contextual data indicates that the previous interaction occurred in a prior context that included one or more features that are shared with a current context in which the user provided the spoken utterance, and
wherein causing the separate speech recognition hypothesis to be processed using the one or more trained machine learning models includes causing the contextual data to be processed using the one or more trained machine learning models and along with the separate speech recognition hypothesis.

### Claim 17 (depends on 16)

The method of claim 16, wherein the one or more features include a location of the computing device in the prior context and the current context.

### Claim 18 (depends on 16)

The method of claim 16, wherein the misrecognition threshold or the alternate threshold are dynamically adjusted based on one or more features of a context of the user.

### Claim 19

A method implemented by one or more processors, the method comprising:
processing audio data capturing a spoken utterance provided by a user to an automated assistant that is accessible via a computing device,
wherein the spoken utterance embodies a request to cause the automated assistant to perform one or more operations in furtherance of fulfilling the request, and
wherein the audio data is processed in furtherance of generating a speech recognition hypothesis that is a prediction of the spoken utterance;
processing input data corresponding to an additional input provided by the user to the automated assistant subsequent to the user providing the spoken utterance to the automated assistant;
processing contextual data corresponding to a context of the user when the user provided the spoken utterance and the additional input to the automated assistant,
wherein the contextual data is provided by the computing device, and/or a separate computing device that is associated with the automated assistant;
determining, based on the audio data, the input data, and the contextual data, a probability metric that indicates a degree to which the additional input is predicted to correct one or more hypothesis terms of the speech recognition hypothesis generated by the automated assistant; and
when the probability metric satisfies a misrecognition threshold:
causing a separate speech recognition hypothesis to be determined for indicating the request to be fulfilled by the automated assistant,
wherein the separate speech recognition hypothesis is a separate prediction of the spoken utterance.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
when the probability metric satisfies the misrecognition threshold:
causing a misrecognition model to be further trained using training data that is based on the audio data, input data, and/or the contextual data.
