Classifier readiness and maintenance in automatic defect classification

A non-transitory computer readable storage medium having instructions that, when executed by a processing device, cause the processing device to perform operations comprising:

• receiving, by a computer system, inspection data associated with a plurality of defects found in one or more samples;
• receiving, by the computer system, one or more benchmark classification results comprising a class for each of the plurality of defects;
• evaluating, by the computer system, a readiness criterion for one or more of the classes based on the one or more benchmark classification results, wherein the readiness criterion comprises for each class, a suitability of the inspection data for training an automatic defect classifier for the class;
• selecting a portion of the inspection data corresponding to one or more defects associated with one or more classes that satisfy the readiness criterion;
• and
• training one or more automatic classifiers for the one or more classes that satisfy the readiness criterion using the selected portion of the inspection data.