Training and testing machine learning models using biased samples

A computer-implemented method comprising:

• receiving data to be annotated;
• receiving annotations associated with the data, wherein at least a proper subset of the annotations is human-generated using the data according to an opt-in preference;
• sampling the annotations to generate a sampled test set by:
• estimating, by an audio annotation service implemented as audio annotation service code executed out of memory by one or more processors, a propensity score for each annotation, wherein the propensity score quantifies a likelihood of each annotation being human-generated using the data,
• estimating a confidence score for each annotation, wherein the confidence score quantifies a confidence in a correctness of the annotation,
• mapping each annotation to a multi-dimensional space based at least in part on the propensity score to generate mapped annotations,
• stratifying, based on the propensity score, the mapped annotations into strata, and
• sampling each stratum of the mapped annotations according to a request to generate the sampled test set;
• and
• performing at least one of evaluating a machine learning (ML) model using the sampled test set or training another ML model using the sampled test set.

A system comprising:

• a first one or more electronic devices to implement a data storage service in a multi-tenant provider network;
• and
• a second one or more electronic devices to implement an annotation service in the multi-tenant provider network, the annotation service including instructions that upon execution cause the annotation service to:
• receive data to be annotated from the data storage service;
• receive annotations associated with the data, wherein at least a proper subset of the annotations is human-generated using the data according to an opt-in preference;
• sample the annotations to generate a sampled test set by
• estimating, by an audio annotation service implemented as audio annotation service code executed out of memory by one or more processors, a propensity score for each annotation, wherein the propensity score quantifies a likelihood of each annotation being human-generated using the data,
• estimating a confidence score for each annotation, wherein the confidence score quantifies a confidence in a correctness of the annotation,
• mapping each annotation to a multi-dimensional space based at least in part on the propensity score to generate mapped annotations,
• stratifying, based on the propensity score, the mapped annotations into strata, and
• sampling each stratum of the mapped annotations according to a request to generate the sampled test set;
• and
• perform at least one of evaluating a machine learning (ML) model using the sampled test set or training another ML model using the sampled test set.