# US 2015/0149176 A1 · System and method for training a classifier for natural language understanding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0149176 A1
- Document ID: US20150149176A1
- Status: Published application
- Kind: A1
- Application number: 14/092,258
- Filed: 2013-11-27
- Published: 2015-05-28
- Earliest claimed: 2013-11-27
- Assignees: AT&T Intellectual Property I, L.P.
- Applicants: AT&T Intellectual Property I, L.P.
- Inventors: Danilo GIULIANELLI; Patrick Guy HAFFNER
- Web page: https://patentgrep.com/patent/US20150149176A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed herein are systems, methods, and computer-readable storage devices for building classifiers in a semi-supervised or unsupervised way. An example system implementing the method can receive a human-generated map which identifies categories of transcriptions. Then the system can receive a set of machine transcriptions. The system can process each machine transcription in the set of machine transcriptions via a set of natural language understanding classifiers, to yield a machine map, the machine map including a set of classifications and a classification score for each machine transcription in the set of machine transcriptions. Then the system can generate silver annotated data by combining the human-generated map and the machine map. The algorithm can include different branches for when the machine transcription is available, when partial results are available, when no results are found for the machine transcription, and so forth.

## Claims

### Claim 1

A method comprising:
receiving a map which identifies categories of human transcribed utterances;
receiving a plurality of machine generated transcriptions;
processing each machine transcription in the plurality of machine transcriptions via a plurality of natural language understanding classifiers, to yield a machine map, the machine map comprising a plurality of classifications and a classification score for each machine transcription in the plurality of machine transcriptions; and
generating, via a processor, silver annotated data by combining the map and the machine map.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of machine transcriptions is generated using a plurality of distinct automatic speech recognizers.

### Claim 3 (depends on 2)

The method of claim 2, wherein the combining comprises:
(1) when a machine transcription is found in the map, adding the machine transcription and an associated category to the silver annotated data;
(2) when the machine transcription is not found in the map, performing a partial match of weighted words in the machine transcription to words in the map, and upon finding a match above a threshold similarity, adding the match and an associated category to the silver annotated data;
(3) when the partial match yields no results for the machine transcription, search each of the plurality of natural language classifiers for the machine transcription and, upon finding matching machine transcriptions and corresponding category in multiple natural language classifiers, adding the matching machine transcription and corresponding category to the silver annotated data; and
(4) when steps (1)-(3) yield no results for the machine transcription, selecting a category corresponding to a natural language classifier in the plurality of natural language classifiers having a highest confidence score associated with a classification, and adding the machine transcription and the category to the silver annotated data.

### Claim 4 (depends on 1)

The method of claim 1, wherein each of the plurality of natural language understanding classifiers is tuned for a different language domain.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
weighting the machine map based on a distance of a respective language domain to a target language domain.

### Claim 6 (depends on 1)

The method of claim 1, wherein the map associates human-generated transcriptions with human-assigned categories.

### Claim 7 (depends on 1)

The method of claim 1, wherein the plurality of machine transcriptions is received as a single list.

### Claim 8

A system comprising:
a processor; and
a computer-readable storage medium storing instructions which, when executed by the processor, cause the processor to perform operations comprising:
receiving a map which identifies categories of transcriptions;
receiving a plurality of machine transcriptions;
processing each machine transcription in the plurality of machine transcriptions via a plurality of natural language understanding classifiers, to yield a machine map, the machine map comprising a plurality of classifications and a classification score for each machine transcription in the plurality of machine transcriptions; and
generating silver annotated data by combining the map and the machine map.

### Claim 9 (depends on 8)

The system of claim 8, wherein the plurality of machine transcriptions is generated using a plurality of distinct automatic speech recognizers.

### Claim 10 (depends on 9)

The system of claim 9, wherein the combining comprises:
(1) when a machine transcription is found in the map, adding the machine transcription and an associated category to the silver annotated data;
(2) when the machine transcription is not found in the map, performing a partial match of weighted words in the machine transcription to words in the map, and upon finding a match above a threshold similarity, adding the match and an associated category to the silver annotated data;
(3) when the partial match yields no results for the machine transcription, search each of the plurality of natural language classifiers for the machine transcription and, upon finding matching machine transcriptions and corresponding category in at least two of the natural language classifiers, adding the matching machine transcription and corresponding category to the silver annotated data; and
(4) when steps (1)-(3) yield no results for the machine transcription, selecting a category corresponding to a natural language classifier in the plurality of natural language classifiers having a highest confidence score associated with a classification, and adding the machine transcription and the category to the silver annotated data.

### Claim 11 (depends on 8)

The system of claim 8, wherein each of the plurality of natural language understanding classifiers is tuned for a different language domain.

### Claim 12 (depends on 11)

The system of claim 11, the computer-readable storage device further storing instructions which result in the method further comprising:
weighting the machine map based on a distance of a respective language domain to a target language domain.

### Claim 13 (depends on 8)

The system of claim 8, wherein the map associates human-generated transcriptions with human-assigned categories.

### Claim 14 (depends on 8)

The system of claim 8, wherein the plurality of machine transcriptions is received as a single list.

### Claim 15

A computer-readable storage device storing instructions which, when executed by a computing device, cause the computing device to perform operations comprising:
receiving a map which identifies categories of transcriptions;
receiving a plurality of machine transcriptions;
processing each machine transcription in the plurality of machine transcriptions via a plurality of natural language understanding classifiers, to yield a machine map, the machine map comprising a plurality of classifications and a classification score for each machine transcription in the plurality of machine transcriptions; and
generating silver annotated data by combining the human generated map and the machine map.

### Claim 16 (depends on 15)

The computer-readable storage device of claim 15, wherein the plurality of machine transcriptions are generated using a plurality of distinct automatic speech recognizers.

### Claim 17 (depends on 16)

The computer-readable storage device of claim 16, wherein the combining comprises:
(1) when a machine transcription is found in the map, adding the machine transcription and an associated category to the silver annotated data;
(2) when the machine transcription is not found in the map, performing a partial match of weighted words in the machine transcription to words in the map, and upon finding a match above a threshold similarity, adding the match and an associated category to the silver annotated data;
(3) when the partial match yields no results for the machine transcription, search each of the plurality of natural language classifiers for the machine transcription and, upon finding matching machine transcriptions and corresponding category in multiple natural language classifiers, adding the matching machine transcription and corresponding category to the silver annotated data; and
(4) when steps (1)-(3) yield no results for the machine transcription, selecting a category corresponding to a natural language classifier in the plurality of natural language classifiers having a highest confidence score associated with a classification, and adding the machine transcription and the category to the silver annotated data.

### Claim 18 (depends on 15)

The computer-readable storage device of claim 15, wherein each of the plurality of natural language understanding classifiers is tuned for a different language domain.

### Claim 19 (depends on 18)

The computer-readable storage device of claim 18, further comprising:
weighting the machine map based on a distance of a respective language domain to a target language domain.

### Claim 20 (depends on 15)

The computer-readable storage device of claim 15, wherein the map associates human-generated transcriptions with human-assigned categories.
