# US 2022/0036883 A1 · Compounding Corrective Actions and Learning in Mixed Mode Dictation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036883 A1
- Document ID: US20220036883A1
- Status: Published application
- Kind: A1
- Application number: 16/942,718
- Filed: 2020-07-29
- Published: 2022-02-03
- Earliest claimed: 2020-07-29
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Emily TRAN; Robert L. CHAMBERS; Erez KIKIN-GIL
- Web page: https://patentgrep.com/patent/US20220036883A1

## CPC

- G10L 15/16: using artificial neural networks
- G06F 40/30: Semantic analysis
- G10L 15/063: {Training}
- G10L 15/065: Adaptation
- G10L 15/18: using natural language modelling
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/0638: {Interactive procedures}
- G10L 2015/221: {Announcement of recognition results}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques performed by a data processing system for processing voice content received from a user herein include receiving a first audio input from the user comprising a mixed-mode dictation, analyzing, using one or more machine learning (ML) models, the first audio input to obtain a first interpretation of the mixed-mode dictation, presenting the first interpretation to the user in an application on the data processing system, receiving a second audio input from the user comprising a corrective command, analyzing the second audio input to obtain a second interpretation of the restatement of the mixed-mode dictation presenting the second interpretation to the user, receiving an indication from the user that the second interpretation is a correct interpretation of the mixed-mode dictation, and modifying the operating parameters of the one or more machine learning models to interpret the subsequent instances of the mixed-mode dictation based on the second interpretation.

## Claims

### Claim 1

A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
receiving a first audio input from a user comprising a mixed-mode dictation, wherein the mixed-mode dictation includes a command to be executed by an application on the data processing system, textual content to be rendered by the application, or both;
analyzing the first audio input to obtain a first interpretation of the mixed-mode dictation by processing the first audio input using one or more natural language processing models to obtain a first textual representation of the mixed-mode dictation and processing the first textual representation using one or more machine learning models to obtain the first interpretation of the mixed-mode dictation;
presenting the first interpretation of the mixed-mode dictation to the user in an application on the data processing system;
receiving a second audio input from the user comprising a corrective command in response to presenting the first interpretation in the application, wherein the second audio input includes a restatement of the mixed-mode dictation with an alternative phrasing;
analyzing the second audio input to obtain a second interpretation of the restatement of the mixed-mode dictation provided by the user by processing the second audio input using one or more natural language processing models to obtain a second textual representation of the restatement of the mixed-mode dictation and processing the second textual representation using one or more machine learning models to obtain the second interpretation of the mixed-mode dictation;
presenting the second interpretation to the user in the application on the data processing system;
receiving an indication from the user that the second interpretation is a correct interpretation of the mixed-mode dictation; and
responsive to the indication from the user, modifying the operating parameters of the one or more machine learning models to interpret the subsequent instances of the mixed-mode dictation based on the second interpretation by generating training data for the one or more machine learning models that associates the mixed-mode dictation with the second interpretation and retraining the one or more machine learning models using the training data.

### Claim 2 (depends on 1)

The data processing system of claim 1, wherein to present the first interpretation to the user in the application includes instructions configured to cause the processor to perform operations of:
processing the first interpretation of the first audio input in the application by executing the command, rendering the textual content, or both included in the mixed-mode dictation.

### Claim 3 (depends on 1)

The data processing system of claim 1, wherein to present the second interpretation to the user in the application includes instructions configured to cause the processor to perform operations of:
processing the second interpretation of the first audio input in an application by executing the command, rendering the textual content, or both included in the mixed-mode dictation.

### Claim 4 (depends on 3)

The data processing system of claim 3, further including instructions configured to cause the processor to perform operations of:
receiving a third audio input from the user comprising the mixed-mode dictation;
analyzing the first audio input using one or more machine learning models to obtain the second interpretation of the mixed-mode dictation; and
processing the third interpretation of the third audio input in an application by executing the command, rendering the textual content, or both included in the mixed-mode dictation.

### Claim 5 (depends on 4)

The data processing system of claim 4, further including instructions configured to cause the processor to perform an operation of:
identifying a term or phrase included in the textual representation of the second audio input indicative of the second audio input including a corrective command, and
wherein to analyze the first textual output using the one or more machine learning models further comprises instructions configured to cause the processor to perform an operation of analyzing the first textual output and the identified term or phrase indicative of the first audio input including a corrective command.

### Claim 6 (depends on 1)

The data processing system of claim 1, wherein to modify the operating parameters of the one or more machine learning models to interpret the subsequent instances of the mixed-mode dictation based on the second interpretation the computer-readable medium includes instructions configured to cause the processor to perform operations of:
updating one or more operating parameters of one or more machine learning models based on a difference between the first interpretation and the second interpretation of the first audio input.

### Claim 7 (depends on 1)

The data processing system of claim 1, wherein to modify the operating parameters of the one or more machine learning models to interpret the subsequent instances of the mixed-mode dictation based on the second interpretation the computer-readable medium includes instructions configured to cause the processor to perform operations of:
generating training data for the one or more machine learning models based on a difference between the first interpretation and the second interpretation of the first audio input.

### Claim 8 (depends on 1)

The data processing system of claim 1, further including instructions configured to cause the processor to perform operations of:
sending one or more commands to an application to undo one or more previous commands responsive to receiving the second audio input comprising the corrective command.

### Claim 9

A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
receiving a first audio input comprising spoken content that includes a first command;
analyzing the first audio input using one or more machine learning models to obtain a first interpretation of the first audio input, the first interpretation including first context information;
processing the first interpretation of the first audio input in an application based on the first context information;
receiving, subsequent to the first audio input, a second audio input comprising spoken content that includes a second command;
analyzing the second audio input using the one or more machine learning models to obtain a second interpretation of the second audio input, the second interpretation including second context information, wherein the second interpretation is based at least in part on the first context information associated with the first command; and
processing the second interpretation in the application based on the second context information.

### Claim 10 (depends on 9)

The data processing system of claim 9, further including instructions configured to cause the processor to perform operations of:
analyzing the first audio input using one or more natural language processing models to produce a first textual output comprising a first textual representation of the first audio input; and
analyzing the first textual output using one or more machine learning models to determine first context information of the first textual output.

### Claim 11 (depends on 10)

The data processing system of claim 10, further including instructions configured to cause the processor to perform operations of:
analyzing the second audio input using one or more natural language processing models to produce a second textual output comprising a second textual representation of the second audio input;
identifying a term or phrase included in the textual representation of the second audio input indicative of the second audio input including a sequential command; and
analyzing the second textual output and the identified term or phrase using one or more machine learning models to determine second context information of the second textual output.

### Claim 12 (depends on 9)

The data processing system of claim 9, further including instructions configured to cause the processor to perform operations of:
receiving implicit feedback or explicit feedback regarding the second interpretation of the second audio input; and
training the one or more machine learning models based on the implicit feedback or the explicit feedback.

### Claim 13 (depends on 12)

The data processing system of claim 12, wherein to the at least one machine learning model based on the implicit feedback or the explicit feedback the computer-readable medium includes instructions configured to cause the processor to perform operations of:
updating at one or more parameters of one or more machine learning models based on a difference between the first interpretation and the second interpretation of the first audio input.

### Claim 14 (depends on 12)

The data processing system of claim 12, wherein to train the at least one machine learning model based on the implicit feedback or the explicit feedback the computer-readable medium includes instructions configured to cause the processor to perform operations of:
generating training data for the one or more machine learning models based on a difference between the first interpretation and the second interpretation of the first audio input.

### Claim 15

A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
receiving a first audio input from a user comprising a spoken content;
identifying a forced-action command and command information included in the spoken content, wherein the forced-action command indicates that a particular action is to be performed based on the command information;
analyzing the first audio input using one or more machine learning models to obtain a first interpretation of the first audio input including first context information, wherein the one or more machine learning models are configured to receive the first audio input and a textual representation of the force-action command and the command information, wherein the first context information includes an interpretation of how the forced-action command is to be applied by an application based on the command information;
processing the first interpretation of the first audio input in an application based on the first context information.

### Claim 16 (depends on 15)

The data processing system of claim 15, further including instructions configured to cause the processor to perform operations of:
analyzing the first audio input using one or more natural language processing models to produce a first textual output comprising a first textual representation of the first audio input.

### Claim 17 (depends on 15)

The data processing system of claim 15, wherein to identify the forced-action command and the command information the computer-readable medium includes instructions to cause the processor to perform operations of:
identifying a term or phrase included in the textual representation of the first audio input indicative of the first audio input including the forced-action command; and wherein to analyze the first audio input and the indication that first audio input includes the forced-action command; and
analyzing the first textual output and the identified term or phrase using one or more machine learning models to determine first context information of the first textual output.

### Claim 18 (depends on 15)

The data processing system of claim 15, further including instructions configured to cause the processor to perform operations of:
receiving implicit feedback or explicit feedback regarding the first interpretation of the first audio input; and
training one or more one machine learning models based on the implicit feedback or the explicit feedback.

### Claim 19 (depends on 18)

The data processing system of claim 18, wherein to the at least one machine learning model based on the implicit feedback or the explicit feedback the computer-readable medium includes instructions configured to cause the processor to perform operations of:
updating at one or more parameters of one or more machine learning models based on a difference between the first interpretation and the implicit feedback or explicit feedback.

### Claim 20 (depends on 18)

The data processing system of claim 18, wherein to train the at least one machine learning model based on the implicit feedback or the explicit feedback the computer-readable medium includes instructions configured to cause the processor to perform operations of:
generating training data for the one or more machine learning models based on a difference between the first interpretation and the implicit feedback or explicit feedback.
