Classifier validation
AssigneeYahoo Assets LLC
InventorsEric Theodore Bax, Natalie Bax
Life of this patent
Estimating the term…
Abstract
One or more computing devices, systems, and/or methods for classifier validation are provided. A set of in-sample examples are partitioned into a reduced in-sample set and a remaining in-sample set. The reduced in-sample set is processed using a set of classifiers. A subset of classifiers are identified as having error counts, over the reduced in-sample set, below a threshold number of errors. A training procedure is executed to select a classifier having a minimum error rate over the set of in-sample examples. If the classifier is within the subset of classifiers, then an out-of-sample error bound is determined for the classifier.
Claims20
A method, comprising:
- executing, on a processor of a computing device, instructions that cause the computing device to perform operations, the operations comprising:
- identifying a subset of classifiers of a set of classifiers as having error counts below a threshold number of errors;
- executing a training procedure to select a classifier from the set of classifiers based upon the classifier having an error rate less than an error rate threshold;
- and
- upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier.
A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising:
- identifying a subset of classifiers of a set of classifiers as having error counts below a threshold number of errors;
- executing a training procedure to select a classifier from the set of classifiers based upon the classifier having a minimum error rate;
- and
- upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier, a count of remaining examples, and a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers.
A computing device comprising:
- a processor;
- and
- memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising:
- determining an upper bound on a probability that a classifier drawn according to a distribution has an error count below a threshold number of errors;
- performing a training procedure for weighted ensemble classifiers to select a posterior distribution over a set of classifiers;
- and
- computing an out-of-sample error bound based upon a divergence between the distribution and the posterior distribution.
Dependencies are inferred from the claim text.