# US 2009/0276384 A1 · Methods and Devices Relating to Estimating Classifier Performance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2009/0276384 A1
- Document ID: US20090276384A1
- Status: Published application
- Kind: A1
- Application number: 12/502,979
- Filed: 2009-07-14
- Published: 2009-11-05
- Earliest claimed: 2002-03-28
- Assignees: Exagen Diagnostics, Inc.
- Inventors: Cole Coryell Harris
- Web page: https://patentgrep.com/patent/US20090276384A1

## CPC

- G16H 10/40: for data related to laboratory analysis, e.g. patient specimen analysis
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24: Classification techniques
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and devices, including methods and devices for estimating classifier performance such as generalization performance, are disclosed. One method includes providing multiple samples. Each sample is characterized by one or more features. This method also includes associating a feature variability with at least one of the one or more features on a feature-by-feature basis; and computing a first probability of misclassification by a first classifier using the feature variability. Devices, including integrated circuits (ICs) and field programmable gate arrays (FPGAs), that are configured for use in carrying out the present methods are also disclosed.

## Claims

### Claim 1

A computer-implemented method for estimating performance of a classifier comprising:
using suitably programmed hardware to associate a feature variability with at least one of one or more features that characterize a sample from multiple samples, each of which is characterized by one or more features;
using the classifier to compute, with the hardware, one or more scores for one or more samples in a subset of samples from the multiple samples, where the classifier can be used to determine a class to which a sample belongs based on the one or more features that characterize that sample;
using the hardware to associate a score variability with at least one of the one or more scores using the feature variability; and
computing, with the hardware, a probability of misclassification by the classifier using one or more of the one or more scores and the score variability.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, where the multiple samples are multiple tissue samples.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, where the classifier is a linear discriminant function classifier.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, where the classifier is a quadratic discriminant function classifier.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, where the classifier is a neural network classifier.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, where the feature variability is estimated using multiple measurements of one of the one or more features.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, where the using hardware to associate a score variability with at least one of the one or more scores includes using a linear approximation of the classifier.

### Claim 8 (depends on 1)

A physical computer readable medium comprising machine-readable instructions for implementing the computer-implemented method of claim 1.

### Claim 9

A computer-implemented method comprising:
using suitably programmed hardware to associate a feature variability with at least one of one or more features on a feature-by-feature basis, the one or more features characterizing a sample from multiple samples, each of which is characterized by one or more features;
computing, with the hardware, a first probability of misclassification by a neural network classifier using the feature variability; and
computing, with the hardware, a first probability of misclassification by a classifier using the feature variability, where the classifier can be used to determine a class to which a sample belongs based on the one or more features that characterize that sample.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, where the multiple samples are multiple tissue samples.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, where the feature variability is estimated using multiple measurements of one of the one or more features.

### Claim 12

A computer-implemented method comprising:
using suitably programmed hardware to associate a feature variability with at least one of one or more features on a feature-by-feature basis, the one or more features characterizing a sample from multiple samples, each of which is characterized by one or more features; and
computing, with the hardware, a first probability of misclassification by a first classifier using the feature variability, where the first classifier can be used to determine a class to which a sample belongs based on the one or more features that characterize that sample;
where the computing, with the hardware, a first probability of misclassification by a first classifier includes using a linear approximation of the first classifier prior to compute the first probability of misclassification.

### Claim 13 (depends on 9, 10, 11)

A physical computer readable medium comprising machine-readable instructions for implementing the computer-implemented method of any of claims 9, 10, 11, and 12.

### Claim 14

A computer-implemented method for estimating performance of a classifier comprising:
using suitably programmed hardware to associate a feature variability with at least one of one or more features that characterize a sample from multiple samples, each of which is characterized by one or more features;
using a first classifier to compute, with the hardware, one or more first scores for one or more first samples in a subset of samples from the multiple samples, where the first classifier can be used to determine a class to which a sample belongs based on the one or more features that characterize that sample;
using the hardware to associate a first score variability with at least one of the one or more first scores using the feature variability;
computing, with the hardware, a first probability of misclassification by the first classifier using one or more of the one or more first scores and the first score variability;
using a second classifier to compute, with the hardware, one or more second scores for one or more second samples in a subset of samples from the multiple samples, where the second classifier can be used to determine a class to which a sample belongs based on the one or more features that characterize that sample;
using the hardware to associate a second score variability with at least one of the one or more second scores using the feature variability;
computing, with the hardware, a second probability of misclassification by the second classifier using one or more of the one or more second scores and the second score variability; and
determining, with the hardware, a final probability of misclassification from a group of probabilities of misclassification that includes the first and second probabilities of misclassification.

### Claim 15 (depends on 1, 2, 3, 4, 5, 6, 7)

A physical computer readable medium comprising machine-readable instructions for implementing the computer-implemented method of any of claims 1-7.

### Claim 16 (depends on 14)

A physical computer readable medium comprising machine-readable instructions for implementing the computer-implemented method of claim 14.