Meta-learning data augmentation framework
AssigneeRecruit Co., Ltd.
InventorsYuliang Li, Xiaolan Wang, Zhengjie Miao
Life of this patent
Estimating the term…
Abstract
Disclosed embodiments relate to generating training data for a machine learning model. Techniques can include accessing a machine learning model from a machine learning model repository and identifying a data set associated with the machine learning model. The identified data set is utilized to generate a set of data augmentation operators. The data augmentation operators applied on a selected sequence of tokens associated with the machine learning model to generate sequences of tokens. A subset of sequences of tokens are selected and stored in a training data repository. The stored sequences of tokens are provided to the machine learning model as training data.
Claims20
A non-transitory computer readable storage medium storing instructions that are executable by a data augmentation system that includes one or more processors to cause the data augmentation system to perform a method for generating training data for a machine learning model, the method comprising:
- accessing a first machine learning model from a machine learning model repository;
- identifying a data set associated with the first machine learning model, wherein the data set is determined based on a task associated with the first machine learning model;
- generating a set of data augmentation operators using the data set, wherein the set of data augmentation operators are configured to adapt transformation operations based on the task associated with the first machine learning model and prior performance data associated with the first machine learning model;
- serializing the data set into serialized token sequences using markers associated with the first machine learning model;
- selecting a first sequence of tokens and a second sequence of tokens from the serialized token sequences to represent the data set;
- selecting a first data augmentation operator from the set of data augmentation operators;
- generating a first additional sequence of tokens by applying the first data augmentation operator to the first selected sequence of tokens;
- generating a second additional sequence of tokens by applying the first data augmentation operator to the second selected sequence of tokens, wherein operations performed in generating the first and the second additional sequences are different and determined based on the task associated with the first machine learning model;
- assigning soft labels to one or more of the first and second additional sequence of tokens, wherein the soft labels are based on proximity metrics determined by existing labeled training data;
- executing a filtering machine learning model trained to minimize validation loss of the first machine learning model, wherein the filtering machine learning model is configured to select a first subset of the generated additional sequences of tokens and discard one of the first subset of selected sequences of tokens;
- executing a weighting machine learning model trained to assign weights to the first subset based on an estimated contribution, wherein the estimated contribution is based on the validation loss associated with the first machine learning model;
- selecting a second subset from the weighted first subset based on an output of the executing of the filtering machine learning model or the weighting machine learning model;
- storing the second subset in a training data repository;
- and
- providing the stored second subset as training data to train the first machine learning model.
A non-transitory computer readable storage medium storing instructions that are executable by a data augmentation system that includes one or more processors to cause the data augmentation system to perform a method for generating data augmentation operators to generate augmented sequences of tokens, the method including operations comprising:
- accessing unlabeled data from an unlabeled data repository;
- preparing one or more sequences of tokens of the accessed unlabeled data, wherein the one or more sequences are serialized using markers associated with a target machine learning model;
- transforming the serialized sequences of tokens to generate a first corrupted sequence and a second corrupted sequence by applying a same data augmentation operator, wherein operations in generating the first and the second corrupted sequences are different and determined based on a task associated with the target machine learning model;
- providing as input, by the data augmentation system, the first corrupted sequence and the second corrupted sequence to a trained sequence-to-sequence model of the data augmentation system;
- executing the trained sequence-to-sequence model using one or more processors of the data augmentation system to determine one or more inverse transformation operations that map the first or the second corrupted sequence to the sequence of tokens used to generate the first or the second corrupted sequence, wherein the one or more inverse transformation operations are determined based on prior performance data associated with the target machine learning model;
- generating one or more inverse data augmentation operators based on the determined one or more inverse transformation operations, wherein the one or more inverse data augmentation operators are configured to reverse the first or the second corrupted sequence based on a task associated with the target machine learning model and the prior performance data;
- and
- training the target machine learning model to perform the task using training data generated by applying the one or more inverse data augmentation operators.
A non-transitory computer readable storage medium storing instructions that are executable by a data augmentation system that includes one or more processors to cause the data augmentation system to perform a method for extracting classification information from input data, the method including operations comprising:
- modifying a first machine learning model by adding task-specific layers to generate a modified network, wherein the task specific layers are trained to extract classification signals for a downstream task;
- initializing the modified network of the first machine learning model and the added task-specific layers;
- selecting input data entries by one or more processors of the data augmentation system, wherein selecting includes serializing the input data entries using markers associated with the first machine learning model;
- providing the serialized input data entries to the modified network;
- extracting, by one or more processors of the data augmentation system, classification information using the task-specific layers of the modified network;
- transforming, by the data augmentation system, the serialized input data entries to generate different augmented data entries by applying a same data augmentation operator to the serialized input data entries, wherein operations in generating the different augmented data entries are different and determined based on a task associated with a second machine learning model and prior performance data associated with the second machine learning model;
- filtering the generated different augmented data entries based on the extracted classification information, wherein filtering includes discarding one or more of the augmented data entries based on validation loss metrics associated with the second machine learning model;
- and
- training the second machine learning model using the filtered augmented data entries as training data.
Dependencies are inferred from the claim text.