Back to results
US 11,893,772 B1Granted
Artificial intelligence system with iterative two-phase active learning
AssigneeAmazon Technologies, Inc.
InventorsSedat Gokalp, Tarun Gupta
Life of this patent
Estimating the term…
Abstract
Learning iterations, individual ones of which include a respective bucket group selection phase and a class boundary refinement phase, are performed using a source data set whose records are divided into buckets. In the bucket group selection phase of an iteration, a bucket is selected for annotation based on output obtained from a classification model trained in the class boundary refinement phase of an earlier iteration. In the class boundary refinement phase, records of buckets annotated as positive-match buckets for a target class in the bucket group selection phase are selected for inclusion in a training set for a new version of the model using a model enhancement criterion. The trained version of the model is stored.
Claims20
A system, comprising:
- one or more computing devices;
- wherein the one or more computing devices include instructions that upon execution on or across one or more processors implement cause the one or more computing devices to:
- obtain an indication, via one or more programmatic interfaces, of a source data set, wherein the source data set comprises a plurality of records to be used to train a classification model for one or more target classes;
- divide the source data set into a plurality of buckets, wherein individual ones of the buckets comprise one or more records;
- execute a plurality of two-phase learning iterations on selected computing resources to train successive versions of the classification model using the source data set until a training completion criterion is met, wherein individual ones of the two-phase learning iterations include (a) a bucket group selection phase and (b) a class boundary refinement phase, and the training completion criterion limits execution of training by a computing resource consumption limit or execution time limit,
- wherein the bucket group selection phase comprises identifying one or more positive-match buckets from among unlabeled buckets in the plurality of buckets, wherein respective ones of the positive-match buckets include at least one record labeled by one or more bucket annotators as a member of a target class of the one or more target classes, wherein the bucket group selection phase eliminates one or more other buckets in the plurality of buckets for annotation to avoid wasting computing resources, wherein at least one bucket is selected as a candidate for annotation based at least in part on output obtained from a first version of a classification model generated in the class boundary refinement phase of an earlier two-phase learning iteration and results of a search query;
- and
- wherein the class boundary refinement phase comprises:
- selecting, from among unlabeled records of the one or more positive-match buckets identified in the bucket group selection phase, a set of labeling-candidate records which meet a model enhancement potential criterion;
- obtaining, from one or more record annotators, class labels for the set of labeling-candidate records;
- and
- training, using a training data set comprising at least the labeled records obtained from the one or more record annotators, a second version of the classification model;
- stop execution of the training when the training completion criterion is met and store a trained version of the classification model produced by the training;
- and
- run the trained version of the classification model to classify one or more records which are not in the source data set.
12 more elements
A method, comprising:
- performing, at one or more computing devices:
- executing a plurality of multi-phase learning iterations to train successive versions of a classification model on selected computing resources using a source data set of a plurality of records until a training completion criterion is met, wherein individual ones of the multi-phase learning iterations include at least (a) a bucket group selection phase and (b) a class boundary refinement phase, and the training completion criterion limits execution of training by a computing resource consumption limit or execution time limit,
- wherein the bucket group selection phase comprises identifying one or more positive-match buckets from among unlabeled buckets in a plurality of buckets, wherein individual ones of the buckets comprise a respective subset of the source data set, wherein respective ones of the positive-match buckets include one or more records labeled as members of a target class of one or more target classes, wherein the bucket group selection phase eliminates one or more other buckets in the plurality of buckets for annotation to avoid wasting computing resources, wherein at least one bucket is selected as a candidate for annotation as a positive-match bucket based at least in part on output obtained from a first version of the classification model generated in the class boundary refinement phase of an earlier learning iteration;
- and
- wherein the class boundary refinement phase comprises training a second version of the classification model using a training data set including at least a subset of records of the positive-match buckets identified in the bucket group selection phase, wherein at least one record of the subset is identified as a candidate for inclusion in the training data set based at least in part on a model enhancement potential criterion;
- and
- stopping execution of the training when the training completion criterion is met and storing a trained version of the classification model produced by the training.
4 more elements
One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause one or more computer systems to:
- perform a plurality of multi-phase learning iterations on selected computing resources to train successive versions of a classification model using a source data set of a plurality of records until a training completion criterion is met, wherein individual ones of the multi-phase learning iterations include at least (a) a bucket group selection phase and (b) a class boundary refinement phase, and the training completion criterion limits execution of training by a computing resource consumption limit or execution time limit,
- wherein the bucket group selection phase comprises identifying one or more positive-match buckets from among unlabeled buckets in a plurality of buckets, wherein individual ones of the buckets comprise a respective subset of the source data set, wherein respective ones of the positive-match buckets include one or more records annotated as members of a target class of one or more target classes, wherein the bucket group selection phase eliminates one or more other buckets in the plurality of buckets for annotation to avoid wasting computing resources, wherein at least one bucket is selected as a candidate for annotation as a positive-match bucket based at least in part on output obtained from a first version of the classification model generated in the class boundary refinement phase of an earlier learning iteration;
- and
- wherein the class boundary refinement phase comprises training a second version of the classification model using a training data set including at least a subset of records of the positive-match buckets identified in the bucket group selection phase, wherein at least one record of the subset is identified as a candidate for inclusion in the training data set based at least in part on a model enhancement potential criterion;
- and
- stop execution of the training when the training completion criterion is met and store a trained version of the classification model produced by the training.
3 more elements
Dependencies are inferred from the claim text.