# US 2013/0254153 A1 · Techniques for evaluation, building and/or retraining of a classification model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0254153 A1
- Document ID: US20130254153A1
- Status: Published application
- Kind: A1
- Application number: 13/429,041
- Filed: 2012-03-23
- Published: 2013-09-26
- Earliest claimed: 2012-03-23
- Assignees: Nuance Communications, Inc.
- Inventors: Etienne Marcheret
- Web page: https://patentgrep.com/patent/US20130254153A1

## CPC

- G06N 20/00: Machine learning
- G06N 7/00: Computing arrangements based on specific mathematical models
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques for evaluation and/or retraining of a classification model built using labeled training data. In some aspects, a classification model having a first set of weights is retrained by using unlabeled input to reweight the labeled training data to have a second set of weights, and by retraining the classification model using the labeled training data weighted according to the second set of weights. In some aspects, a classification model is evaluated by building a similarity model that represents similarities between unlabeled input and the labeled training data and using the similarity model to evaluate the labeled training data to identify a subset of the plurality of items of labeled training data that is more similar to the unlabeled input than a remainder of the labeled training data.

## Claims

### Claim 1

A method of processing a first classification model that classifies an input into one of a plurality of classes, wherein the first classification model was built using labeled training data having a first set of weights, wherein each item of labeled training data is labeled with one of the plurality of classes, the method comprising acts of:
(A) obtaining unlabeled input for the first classification model;
(B) using the unlabeled input to reweight the labeled training data to have a second set of weights that is different from the first set of weights; and
(C) using the labeled training data reweighted according to the second set of weights to:
retrain the first classification model;
and/or
build a second classification model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the act (B) comprises using the unlabeled input to reweight the labeled training data by changing weight values for at least a subset of the labeled training data.

### Claim 3

A method for use with a first classification model that classifies an input into one of a plurality of classes, wherein the first classification model was built using labeled training data, wherein the labeled training data comprises a plurality of items of labeled training data, wherein each of the plurality of items of labeled training data is labeled with one of the plurality of classes, the method comprising acts of:
obtaining unlabeled input for the first classification model;
building a similarity model that represents similarities between the unlabeled input and the labeled training data; and
using a programmed processor and the similarity model to evaluate the labeled training data to identify a subset of the plurality of items of labeled training data that is more similar to the unlabeled input than a remainder of the labeled training data.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
providing the subset of the plurality of items of labeled training data as input to the first classification model;
presenting results of the first classification model in processing the subset of the plurality of items of labeled training data as input.

### Claim 5 (depends on 3)

The method of claim 3, wherein using the programmed processor and the similarity model to evaluate the labeled training data further comprises:
using the similarity model to perform a transformation of a feature space of the labeled training data to create transformed labeled training data; and
processing the transformed labeled training data using the first classification model.

### Claim 6 (depends on 5)

The method of claim 5, wherein the transformation of the feature space is based on desired performance characteristics of the classification model.

### Claim 7 (depends on 6)

The method of claim 6, wherein a first class of the plurality of classes is an accept class, and wherein the desired performance characteristics comprise a desired rate for falsely classifying inputs as belonging to the accept class by the classification model.

### Claim 8 (depends on 7)

The method of claim 7, wherein a second class of the plurality of classes is a reject class, and wherein the desired performance characteristics further comprise a desired rate for falsely classifying inputs as belonging to the reject class by the classification model.

### Claim 9 (depends on 3)

The method of claim 3, further comprising:
using the similarity model to identify one or more items of unlabeled input that are less similar to the labeled training data than a remainder of the unlabeled input;
obtaining labels for the one or more items of unlabeled input that are less similar to the labeled training data to create one or more items of labeled test input; and
using the one or more items of labeled test input and at least a portion of the plurality of items of labeled training data to:
retrain the first classification model;
and/or
build a second classification model.

### Claim 10 (depends on 3)

The method of claim 3, further comprising:
using the subset of the plurality of items of labeled training data that is more similar to the unlabeled input than a remainder of the labeled training data to:
retrain the first classification model;
and/or
build a second classification model.

### Claim 11

A tangible computer-recordable medium having a plurality of instructions embodied therein, wherein the plurality of instructions, when executed by a processor, cause a machine to perform a method of processing a first classification model that classifies an input into one of a plurality of classes, wherein the first classification model was built using labeled training data having a first set of weights, wherein each item of labeled training data is labeled with one of the plurality of classes, the method comprising acts of:
(A) obtaining unlabeled input for the first classification model;
(B) using the unlabeled input to reweight the labeled training data to have a second set of weights that is different from the first set of weights; and
(C) using the labeled training data reweighted according to the second set of weights to:
retrain the first classification model;
and/or
build a second classification model.

### Claim 12 (depends on 11)

The tangible computer-recordable medium of claim 11, wherein the act (B) comprises using the unlabeled input to reweight the labeled training data by changing weight values for at least a subset of the labeled training data.

### Claim 13

A tangible computer-recordable medium having a plurality of instructions embodied therein, wherein the plurality of instructions, when executed by a processor, cause a machine to perform a method for use with a first classification model that classifies an input into one of a plurality of classes, wherein the first classification model was built using labeled training data, wherein the labeled training data comprises a plurality of items of labeled training data, wherein each of the plurality of items of labeled training data is labeled with one of the plurality of classes, the method comprising acts of:
obtaining unlabeled input for the first classification model;
building a similarity model that represents similarities between the unlabeled input and the labeled training data; and
using a programmed processor and the similarity model to evaluate the labeled training data to identify a subset of the plurality of items of labeled training data that is more similar to the unlabeled input than a remainder of the labeled training data.

### Claim 14 (depends on 13)

The tangible computer-recordable medium of claim 13, wherein the method further comprises:
providing the subset of the plurality of items of labeled training data as input to the first classification model;
presenting results of the first classification model in processing the subset of the plurality of items of labeled training data as input.

### Claim 15 (depends on 13)

The tangible computer-recordable medium of claim 13, wherein using the programmed processor and the similarity model to evaluate the labeled training data further comprises:
using the similarity model to perform a transformation of a feature space of the labeled training data to create transformed labeled training data; and
processing the transformed labeled training data using the first classification model.

### Claim 16 (depends on 15)

The tangible computer-recordable medium of claim 15, wherein the transformation of the feature space is based on desired performance characteristics of the classification model.

### Claim 17 (depends on 16)

The tangible computer-recordable medium of claim 16, wherein a first class of the plurality of classes is an accept class, and wherein the desired performance characteristics comprise a desired rate for falsely classifying inputs as belonging to the accept class by the classification model.

### Claim 18 (depends on 17)

The tangible computer-recordable medium of claim 17, wherein a second class of the plurality of classes is a reject class, and wherein the desired performance characteristics further comprise a desired rate for falsely classifying inputs as belonging to the reject class by the classification model.

### Claim 19 (depends on 13)

The tangible computer-recordable medium of claim 13, wherein the method further comprises:
using the similarity model to identify one or more items of unlabeled input that are less similar to the labeled training data than a remainder of the unlabeled input;
obtaining labels for the one or more items of unlabeled input that are less similar to the labeled training data to create one or more items of labeled test input; and
using the one or more items of labeled test input and at least a portion of the plurality of items of labeled training data to:
retrain the first classification model;
and/or
build a second classification model.

### Claim 20 (depends on 13)

The tangible computer-recordable medium of claim 13, wherein the method further comprises:
using the subset of the plurality of items of labeled training data that is more similar to the unlabeled input than a remainder of the labeled training data to:
retrain the first classification model;
and/or
build a second classification model.
