# US 2009/0276380 A1 · Computer-aided natural language annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2009/0276380 A1
- Document ID: US20090276380A1
- Status: Published application
- Kind: A1
- Application number: 12/436,387
- Filed: 2009-05-06
- Published: 2009-11-05
- Earliest claimed: 2002-05-10
- Assignees: Microsoft Corporation
- Inventors: Alejandro Acero; Ye-Yi Wang; Leon Wong
- Web page: https://patentgrep.com/patent/US20090276380A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present invention uses a natural language understanding system that is currently being trained to assist in annotating training data for training that natural language understanding system. Unannotated training data is provided to the system and the system proposes annotations to the training data. The user is offered an opportunity to confirm or correct the proposed annotations, and the system is trained with the corrected or verified annotations.

## Claims

### Claim 1

A method, performed using one or more processors, of generating annotated training data for training a natural language understanding system, comprising:
generating, with the natural language understanding system running on one or more of the processors, a proposed annotation for each of a plurality of units of unannotated training data received via one or more input components;
calculating, with one or more of the processors, a confidence measure for each of the proposed annotations for a given unit of the training data;
displaying on an output component at least some of the proposed annotations in an order based on the confidence measures of the proposed annotations, with one or more input components providing one or more user-actuable inputs configured for verification of the proposed annotations and one or more user-actuable inputs configured for deletion of the proposed annotations;
responding, with one or more of the processors, to an input for verification of one of the proposed annotations by storing the verified annotation for the given unit of the training data; and
responding, with one or more of the processors, to an input for deletion of one of the proposed annotations by presenting on an output component at least some of the remaining proposed annotations in an order based on the confidence measures of the remaining proposed annotations.

### Claim 2 (depends on 1)

The method of claim 1 in which displaying comprises:
displaying the proposed annotations in order based on ascending value of the corresponding confidence measures.

### Claim 3 (depends on 1)

The method of claim 1 in which displaying comprises:
displaying the proposed annotations in order based on descending value of the corresponding confidence measures.

### Claim 4 (depends on 1)

The method of claim 1 in which re-displaying comprises:
displaying the user-confirmed annotations by visually contrasting portions in which the NLU system has a calculated confidence measure below a threshold.

### Claim 5 (depends on 1)

The method of claim 1 and further comprising:
training the NLU system with the user-selected annotation.

### Claim 6

A method, performed using one or more processors, of generating annotated training data for training a natural language understanding (NLU) system, comprising:
generating, with the NLU system running on one or more of the processors, a proposed annotation for each of a plurality of units of unannotated training data, each proposed annotation having a type;
displaying on an output component at least some of the proposed annotations in an order based on the type of each of the proposed annotations, with one or more input components providing one or more user-actuable inputs configured for verification of the proposed annotations and one or more user-actuable inputs configured for deletion of the proposed annotations;
responding, with one or more of the processors, to an input for verification of one of the proposed annotations by storing the verified annotation for the given unit of the training data; and
responding, with one or more of the processors, to an input for deletion of one of the proposed annotations by presenting at least some of the remaining proposed annotations in an order based on the type of each of the remaining proposed annotations.

### Claim 7 (depends on 6)

The method of claim 6 in which displaying comprises:
sorting the proposed annotations based on type; and
displaying the proposed annotations with those of a similar type grouped together.

### Claim 8 (depends on 6)

The method of claim 6 in which re-displaying comprises:
displaying the user-confirmed annotations by visually contrasting portions in which the NLU system has a calculated confidence measure below a threshold.

### Claim 9 (depends on 6)

The method of claim 6 and further comprising:
training the NLU system with the user-verified annotations.

### Claim 10

A method, performed using one or more processors, of generating annotated training data for training a natural language understanding (NLU) system employing a plurality of different natural language training techniques, comprising:
generating, with the natural language understanding system running on one or more of the one or more processors, a different proposed annotation with each of a plurality of the natural language training techniques for a unit of unannotated training data, to obtain a plurality of proposed annotations generated by the natural language training techniques used for that unit of unannotated training data;
displaying on an output component one or more of the proposed annotations, with user actuable inputs for user rejection or user selection of each of the displayed proposed annotations;
responding, with one or more of the one or more processors, to an input for user rejection of one of the proposed annotations by displaying on an output component one or more of any remaining proposed annotations; and
responding, with one or more of the processors, to an input for user selection of one of the proposed annotations by storing the selected annotation for the given unit of the training data.

### Claim 11 (depends on 10)

The method of claim 10 in which displaying comprises:
displaying the proposed annotations with corresponding indications of the natural language training technique with which one or more of the proposed annotations were generated.

### Claim 12 (depends on 10)

The method of claim 10 in which displaying the proposed annotations comprises:
sorting the proposed annotations based on the natural language training techniques with which the proposed annotations were generated; and
displaying the proposed annotations with those generated with the same natural language training techniques grouped together.

### Claim 13 (depends on 10)

The method of claim 10 in which displaying comprises:
displaying the proposed annotations in order based on ascending value of corresponding confidence measures.

### Claim 14 (depends on 10)

The method of claim 10 and further comprising:
training the NLU system with the user-selected annotations.
