# US 2023/0053928 A1 · Classifier validation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0053928 A1
- Document ID: US20230053928A1
- Status: Published application
- Kind: A1
- Application number: 18/045,827
- Filed: 2022-10-11
- Published: 2023-02-23
- Earliest claimed: 2020-09-28
- Assignees: Yahoo Assets LLC
- Applicants: Yahoo Assets LLC
- Inventors: Eric Theodore Bax; Natalie Bax
- Web page: https://patentgrep.com/patent/US20230053928A1

## CPC

- G06F 11/076: {by exceeding a count or rate limit, e.g. word- or bit count limit}
- G06F 11/0727: {in a storage system, e.g. in a DASD or network based storage system (drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)}
- G06F 16/285: {Clustering or classification}
- G06F 16/906: Clustering; Classification
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

One or more computing devices, systems, and/or methods for classifier validation are provided. A set of in-sample examples are partitioned into a reduced in-sample set and a remaining in-sample set. The reduced in-sample set is processed using a set of classifiers. A subset of classifiers are identified as having error counts, over the reduced in-sample set, below a threshold number of errors. A training procedure is executed to select a classifier having a minimum error rate over the set of in-sample examples. If the classifier is within the subset of classifiers, then an out-of-sample error bound is determined for the classifier.

## Claims

### Claim 1

A method, comprising:
executing, on a processor of a computing device, instructions that cause the computing device to perform operations, the operations comprising:
identifying a subset of classifiers of a set of classifiers as having error counts below a threshold number of errors;
executing a training procedure to select a classifier from the set of classifiers based upon the classifier having an error rate less than an error rate threshold; and
upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining the out-of-sample error bound comprises:
determining the out-of-sample error bound based upon a count of remaining examples.

### Claim 3 (depends on 1)

The method of claim 1, wherein the determining the out-of-sample error bound comprises:
determining the out-of-sample error bound based upon a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers.

### Claim 4 (depends on 1)

The method of claim 1, comprising:
reporting a bound failure based upon the classifier not being in the subset of classifiers.

### Claim 5 (depends on 1)

The method of claim 1, wherein the identifying a subset of classifiers comprises:
in response to the set of classifiers exceeding a threshold number of classifiers, sampling a portion of the set of classifiers as a set of sampled classifiers for identifying the subset of classifiers.

### Claim 6 (depends on 5)

The method of claim 5, wherein the identifying a subset of classifiers comprises:
identifying a fraction of the set of sampled classifiers that have error counts below the threshold number of errors.

### Claim 7 (depends on 6)

The method of claim 6, wherein the identifying a subset of classifiers comprises:
utilizing the fraction of the set of sampled classifiers to bound a rate at which classifiers in the set of classifiers have error counts below the threshold number of errors.

### Claim 8 (depends on 1)

The method of claim 1, wherein the identifying a subset of classifiers comprises:
in response to the set of classifiers not exceeding a threshold number of classifiers, testing all classifiers within the set of classifiers for identifying classifiers having error counts below the threshold number of errors.

### Claim 9 (depends on 1)

The method of claim 1, comprising:
in response to the out-of-sample error bound being within a tolerance threshold, utilizing the classifier to classify out-of-sample data.

### Claim 10 (depends on 1)

The method of claim 1, comprising:
in response to the out-of-sample error bound exceeding a tolerance threshold, obtaining additional in-sample examples for training the set of classifiers.

### Claim 11

A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising:
identifying a subset of classifiers of a set of classifiers as having error counts below a threshold number of errors;
executing a training procedure to select a classifier from the set of classifiers based upon the classifier having a minimum error rate; and
upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier, a count of remaining examples, and a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers.

### Claim 12 (depends on 11)

The non-transitory machine readable medium of claim 11, wherein the operations comprise:
in response to the out-of-sample error bound being within a tolerance threshold, utilizing the classifier to classify out-of-sample data.

### Claim 13

A computing device comprising:
a processor; and
memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising:
determining an upper bound on a probability that a classifier drawn according to a distribution has an error count below a threshold number of errors;
performing a training procedure for weighted ensemble classifiers to select a posterior distribution over a set of classifiers; and
computing an out-of-sample error bound based upon a divergence between the distribution and the posterior distribution.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the probability is a shrinkage ratio, and wherein the computing an out-of-sample error bound comprises:
adding a natural logarithm of the upper bound as the shrinkage ratio to the divergence.

### Claim 15 (depends on 13)

The computing device of claim 13, wherein the computing an out-of-sample error bound comprises:
utilizing in-sample examples for computing the out-of-sample error bound.

### Claim 16 (depends on 13)

The computing device of claim 13, wherein the computing an out-of-sample error bound comprises:
adding a fraction of the posterior distribution placed on classifiers that do not have error counts below the threshold number of errors to the out-of-sample error bound.

### Claim 17 (depends on 13)

The computing device of claim 13, wherein the out-of-sample error bound is a PAC-Bayes out-of-sample error bound.

### Claim 18 (depends on 13)

The computing device of claim 13, wherein the out-of-sample error bound is based upon a Kullback-Leibler divergence.

### Claim 19 (depends on 13)

The computing device of claim 13, wherein the operations comprise:
upon a bound being within an acceptance threshold, utilizing the classifier on out-of-sample data.

### Claim 20 (depends on 13)

The computing device of claim 13, wherein the operations comprise:
upon a bound not being within an acceptance threshold, obtaining additional in-sample data.