# US 2014/0278355 A1 · Using human perception in building language understanding models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0278355 A1
- Document ID: US20140278355A1
- Status: Published application
- Kind: A1
- Application number: 13/826,173
- Filed: 2013-03-14
- Published: 2014-09-18
- Earliest claimed: 2013-03-14
- Assignees: Microsoft Corporation
- Applicants: Microsoft Corporation
- Inventors: Ruhi Sarikaya; Anoop Deoras; Fethiye Asli Celikyilmaz; Zhaleh Feizollahi
- Web page: https://patentgrep.com/patent/US20140278355A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/30: Semantic analysis
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An understanding model is trained to account for human perception of the perceived relative importance of different tagged items (e.g. slot/intent/domain). Instead of treating each tagged item as equally important, human perception is used to adjust the training of the understanding model by associating a perceived weight with each of the different predicted items. The relative perceptual importance of the different items may be modeled using different methods (e.g. as a simple weight vector, a model trained using features (lexical, knowledge, slot type, . . . ), and the like). The perceptual weight vector and/or or model are incorporated into the understanding model training process where items that are perceptually more important are weighted more heavily as compared to the items that are determined by human perception as less important.

## Claims

### Claim 1

A method for using human perception in training a language understanding model, comprising:
accessing tagged items that are used by the language understanding model;
creating queries using the tagged items including errors into a portion of the queries that create a query that affects a weight at least one of the tagged items used within the query being created;
obtaining results from a knowledge source for each of the queries created;
obtaining a human perception rating for each of the different results;
determining a relative importance of each of the tagged items using the human perception ratings; and
training the language understanding model using the relative importance of each of the tagged items as determined from the human perception rating.

### Claim 2 (depends on 1)

The method of claim 1, wherein accessing the tagged items that are used by the language understanding model comprises accessing a language understanding corpus that comprises tagged items selected from at least one of: slots;
domains; and intents.

### Claim 3 (depends on 1)

The method of claim 1, wherein creating the queries comprises removing at least one of the tagged items from at least one of the queries created.

### Claim 4 (depends on 1)

The method of claim 1, wherein creating the queries comprises changing a value associated with at least one of the tagged items from at least one of the queries created.

### Claim 5 (depends on 1)

The method of claim 1, wherein obtaining the human perception rating for each of the different result comprises displaying a human perception rating panel that includes a display of: a current query; a current result set; and a human perception rating user interface element that is configured to receive a human perception rating for the result set.

### Claim 6 (depends on 1)

The method of claim 1, wherein obtaining the human perception rating for each of the different result comprises determining a relative importance of each of the tagged items in response to receiving a rating indication for each of the different results.

### Claim 7 (depends on 1)

The method of claim 1, wherein accessing the tagged items that are used by the language understanding model comprises accessing different tagged utterances that are used in training the understanding model.

### Claim 8 (depends on 1)

The method of claim 1, further comprising aggregating human perception ratings obtained from a plurality of judges.

### Claim 9 (depends on 1)

The method of claim 1, wherein training the language understanding model comprises minimizing perceptual error determined from the human perception ratings.

### Claim 10

A computer-readable medium storing computer-executable instructions for using human perception in a language understanding model, comprising:
accessing a language understanding corpus comprising tagged items that are used by the language understanding model including at least one of: tagged slots; tagged domains; and tagged intents;
creating queries using the tagged items including errors into a portion of the queries that create a query that change an effect of the tagged items used within the query being created in obtaining results;
obtaining results from a knowledge source for each of the queries created;
obtaining a human perception rating for each of the different results;
determining a relative importance of each of the tagged items using the human perception ratings; and
training the language understanding model using the relative importance of each of the tagged items as determined from the human perception rating.

### Claim 11 (depends on 10)

The computer-readable medium of claim 10, wherein creating the queries comprises at least one of removing at least one of the tagged items from at least one of the queries created and changing a value associated with at least one of the tagged items from at least one of the queries created.

### Claim 12 (depends on 10)

The computer-readable medium of claim 10, wherein obtaining the human perception rating for each of the different result comprises displaying a human perception rating panel that includes a display of: a current query; a current result set; and a human perception rating user interface element that is configured to receive a human perception rating for the result set.

### Claim 13 (depends on 10)

The computer-readable medium of claim 10, wherein obtaining the human perception rating for each of the different result comprises determining a relative importance of each of the tagged items in response to receiving a rating indication for each of the different results.

### Claim 14 (depends on 10)

The computer-readable medium of claim 10, wherein accessing the tagged items that are used by the language understanding model comprises accessing different tagged utterances that are used in training the language understanding model.

### Claim 15 (depends on 10)

The computer-readable medium of claim 10, further comprising aggregating human perception ratings obtained from a plurality of judges.

### Claim 16 (depends on 10)

The computer-readable medium of claim 10, wherein training the language understanding model comprises minimizing perceptual error determined from the human perception ratings.

### Claim 17

A system for using human perception in a language understanding model, comprising:
a processor and memory;
an operating environment executing using the processor;
a display; and
a perception manager that is configured to perform actions comprising:
accessing a language understanding corpus comprising tagged items that are used by the language understanding model including at least one of: tagged slots; tagged domains; and tagged intents;
creating queries using the tagged items including errors into a portion of the queries that create a query that change an effect of the tagged items used within the query being created in obtaining results;
obtaining results from a knowledge source for each of the queries created;
displaying a human perception rating panel on the display that includes a display of: a current query; a current result set;
and a human perception rating user interface element that is configured to receive a human perception rating for the result set;
obtaining a human perception rating for each of the different results using the human perception rating panel;
aggregating human perception ratings obtained from a plurality of judges;
determining a relative importance of each of the tagged items using the human perception ratings; and
training the language understanding model by minimizing perceptual error determined from the human perception ratings.

### Claim 18 (depends on 17)

The system of claim 17, wherein creating the queries comprises at least one of removing at least one of the tagged items from at least one of the queries created and changing a value associated with at least one of the tagged items from at least one of the queries created.

### Claim 19 (depends on 17)

The system of claim 17, wherein obtaining the human perception rating for each of the different result comprises determining a relative importance of each of the tagged items in response to receiving a rating indication for each of the different results.

### Claim 20 (depends on 17)

The system of claim 17, wherein accessing the tagged items that are used by the language understanding model comprises accessing different tagged utterances that are used in training the language understanding model.
