# US 2019/0244601 A1 · Utterance annotation user interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244601 A1
- Document ID: US20190244601A1
- Status: Published application
- Kind: A1
- Application number: 16/230,536
- Filed: 2018-12-21
- Published: 2019-08-08
- Earliest claimed: 2018-02-05
- Assignees: Semantic Machines, Inc.
- Applicants: Semantic Machines, Inc.
- Inventors: Jesse Daniel Eskes RUSAK; Percy Shuo LIANG
- Web page: https://patentgrep.com/patent/US20190244601A1

## CPC

- G10L 15/063: {Training}
- G10L 15/08: Speech classification or search
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/088: {Word spotting}
- G10L 2015/225: {Feedback of the input speech}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A computing device includes a display configured to present a graphical user interface. The graphical user interface includes a transcript portion configured to display an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, at least one of the dialogue events taking the form of an example client utterance, and an annotation portion configured to display a hierarchical menu including a plurality of candidate utterance annotations. An utterance annotation machine is configured to receive one or more computer inputs selecting, for each of one or more response parameters in the example client utterance, utterance annotations from the hierarchical menu that collectively define a machine-readable interpretation of the example client utterance. An annotated utterance having a predetermined format usable to train the computerized assistant is output to a data store based on the example client utterance

## Claims

### Claim 1

A computing device, comprising:
a display configured to present a graphical user interface, the graphical user interface comprising:
a transcript portion configured to display an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, at least one of the dialogue events taking the form of an example client utterance; and
an annotation portion configured to display a hierarchical menu including a plurality of candidate utterance annotations; and
an utterance annotation machine configured to:
receive one or more computer inputs selecting, for each of one or more response parameters in the example client utterance, one or more utterance annotations from the hierarchical menu, the one or more selected utterance annotations collectively defining a machine-readable interpretation of the example client utterance; and
output, to a data store, an annotated utterance based on the example client utterance and each of the one or more selected utterance annotations, the annotated utterance having a predetermined format usable to train the computerized assistant.

### Claim 2 (depends on 1)

The computing device of claim 1, where the utterance annotation machine is further configured to receive a computer input to select, from the hierarchical menu of candidate utterance annotations, a category-specific domain that exposes a subset of the plurality of candidate utterance annotations related to the category-specific domain.

### Claim 3 (depends on 2)

The computing device of claim 2, where the hierarchical menu of candidate utterance annotations includes a plurality of category-specific domains that are hierarchically nested, such that a particular category-specific domain is associated with one or more category-specific subdomains.

### Claim 4 (depends on 2)

The computing device of claim 2, where the utterance annotation machine is further configured to automatically display a recommendation including one or more suggested category-specific domains that are identified based on detected keywords in the example client utterance.

### Claim 5 (depends on 4)

The computing device of claim 4, where the recommendation further includes one or more suggested candidate utterance annotations identified based on detected keywords in the example client utterance.

### Claim 6 (depends on 1)

The computing device of claim 1, where a dialogue event in the unannotated transcript takes the form of a second example client utterance occurring later in the ordered sequence than the example client utterance, and the second example client utterance refers back to the example client utterance.

### Claim 7 (depends on 6)

The computing device of claim 6, where the utterance annotation machine is further configured to receive, for one or more response parameters in the second example client utterance, one or more computer inputs selecting one or more utterance annotations from the hierarchical menu defining a machine-readable interpretation of the second example client utterance and modifying the one or more selected utterance annotations for the example client utterance.

### Claim 8 (depends on 1)

The computing device of claim 1, where the utterance annotation machine is further configured to, after receiving the one or more computer inputs selecting the one or more utterance annotations, generate and display a computerized assistant response to the example client utterance based on the one or more selected utterance annotations.

### Claim 9 (depends on 8)

The computing device of claim 8, where the utterance annotation machine is further configured to, after generating and displaying the computerized assistant response to the example client utterance, receive one or more computer inputs modifying the one or more selected utterance annotations to address inconsistencies between the example client utterance and the computerized assistant response.

### Claim 10 (depends on 8)

The computing device of claim 8, where generating and displaying the computerized assistant response to the example client utterance includes communicating with a remote service to retrieve information requested in the example client utterance.

### Claim 11 (depends on 1)

The computing device of claim 1, where the utterance annotation machine is further configured to receive one or more computer inputs specifying a computerized assistant response to the example client utterance.

### Claim 12 (depends on 11)

The computing device of claim 11, where the utterance annotation machine is further configured to, based on the one or more selected utterance annotations, automatically generate multiple candidate portions of the computerized assistant response, and where the one or more computer inputs specifying the computerized assistant response select from among the multiple candidate portions.

### Claim 13 (depends on 1)

The computing device of claim 1, where the predetermined format of the annotated utterance is a JavaScript Object Notation (JSON) file.

### Claim 14 (depends on 1)

The computing device of claim 1, where the example client utterance is retrieved from a database of unannotated example client utterances.

### Claim 15

A method, comprising:
graphically displaying an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, at least one of the dialogue events taking the form of an example client utterance;
graphically displaying a hierarchical menu including a plurality of candidate utterance annotations;
receiving one or more computer inputs selecting, for each of one or more response parameters in the example client utterance, one or more utterance annotations from the hierarchical menu, the one or more selected utterance annotations collectively defining a machine-readable interpretation of the example client utterance; and
outputting, to a data store, an annotated utterance based on the example client utterance and each of the one or more selected utterance annotations, the annotated utterance having a predetermined format usable to train the computerized assistant.

### Claim 16 (depends on 15)

The method of claim 15, where the utterance annotation machine is further configured to receive a computer input to select, from the hierarchical menu of candidate utterance annotations, a category-specific domain that exposes a subset of the plurality of candidate utterance annotations related to the category-specific domain.

### Claim 17 (depends on 16)

The method of claim 16, where the hierarchical menu of candidate utterance annotations includes a plurality of category-specific domains that are hierarchically nested, such that a particular category-specific domain is associated with one or more category-specific subdomains.

### Claim 18 (depends on 15)

The method of claim 15, further comprising, after receiving the one or more computer inputs selecting the one or more utterance annotations, generating and displaying a computerized assistant response to the example client utterance based on the one or more selected utterance annotations.

### Claim 19 (depends on 15)

The method of claim 15, further comprising, after generating and displaying the computerized assistant response to the example client utterance, receiving one or more computer inputs modifying the one or more selected utterance annotations to address inconsistencies between the example client utterance and the computerized assistant response.

### Claim 20

A computing device, comprising:
a display engine configured to present a graphical user interface on a display device, the graphical user interface comprising:
a transcript portion configured to display an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, at least one of the dialogue events taking the form of an example client utterance; and
an annotation portion configured to display a hierarchical menu including a plurality of candidate utterance annotations, each of the plurality of candidate utterance annotations being associated with one or more category-specific domains, such that selection of a category-specific domain exposes one or more candidate utterance annotations associated with the selected category-specific domain; and
an utterance annotation machine configured to:
receive one or more computer inputs selecting, for each of one or more response parameters in the example client utterance, one or more utterance annotations from the hierarchical menu, the one or more selected utterance annotations collectively defining a machine-readable interpretation of the example client utterance;
generate and display a computerized assistant response to the example client utterance based on the one or more selected utterance annotations;
receive one or more computer inputs modifying the one or more selected utterance annotations to address inconsistencies between the example client utterance and the computerized assistant response; and
output, to a data store, an annotated utterance based on the example client utterance and each of the one or more selected utterance annotations, the annotated utterance having a predetermined format usable to train the computerized assistant.
