# US 12,118,064 B2 · Training machine learning models using unsupervised data augmentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,064 B2 - Document ID: US12118064B2 - Status: Granted - Kind: B2 - Application number: 17/606,190 - Filed: 2020-04-24 - Published: 2022-07-07 - Granted: 2024-10-15 - Earliest claimed: 2019-04-25 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Thang Minh Luong; Quoc V. Le; Qizhe Xie; Zihang Dai - Web page: https://patentgrep.com/patent/US12118064B2 ## CPC - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a IT machine learning model. One of the methods includes receiving training data comprising a plurality of unlabeled training inputs and a plurality of labeled training inputs; generating augmented training data, comprising generating, for each of the plurality of unlabeled training inputs, a respective augmented training input by applying a data augmentation technique to the unlabeled training input; and training the machine learning model on the augmented training data. In particular, but not exclusively, the model may be trained for perceptual tasks (e.g. tasks relating to vision or speech). ## Claims ### Claim 1 A computer-implemented method comprising: receiving training data for training a machine learning model to map model inputs to model outputs in order to perform a particular machine learning task, the training data comprising: a plurality of unlabeled training inputs; and a plurality of labeled training inputs and, for each labeled training input, a ground truth output that should be generated by the machine learning model by performing the particular machine learning task on the labeled training input; generating augmented training data, comprising generating, for each of the plurality of unlabeled training inputs, a respective augmented training input by applying a data augmentation technique to the unlabeled training input; training the machine learning model on the augmented training data, comprising: training the machine learning model on the unlabeled training inputs and the augmented training inputs to optimize an unsupervised objective that measures a difference between (i) a model output generated by the machine learning model for a given unlabeled training input and (ii) a model output generated by the machine learning model for the augmented training input generated from the unlabeled training input, and training the machine learning model on the labeled training inputs to optimize a supervised objective that measures a difference between (i) a model output generated by the machine learning model for a given labeled training input and (ii) the ground truth output for the given labeled training input. ### Claim 2 (depends on 1) The method of claim 1, wherein the model outputs are probability distributions and wherein the unsupervised objective is based on a K-L divergence between (i) the model output generated by the machine learning model for the given unlabeled training input and (ii) the model output generated by the machine learning model for the augmented training input generated from the unlabeled training input. ### Claim 3 (depends on 1) The method of claim 1, wherein the model outputs are probability distributions and wherein the supervised objective is based on a negative log likelihood of the given ground truth output according to the probability distribution generated for the given labeled training input. ### Claim 4 (depends on 1) The method of claim 1, wherein the labeled and unlabeled training inputs in the augmented training data have been augmented by applying a different data augmentation technique from the data augmentation technique used to generate the augmented unlabeled training inputs. ### Claim 5 (depends on 1) The method of claim 1, wherein the model outputs are probability distributions, wherein the supervised objective is: based on a negative log likelihood of the given ground truth output according to the probability distribution generated for the given labeled training input when the probability assigned to the given ground truth output by the probability distribution is less than a confidence threshold, and equal to zero when the probability assigned to the given ground truth output by the probability distribution is equal to or greater than the confidence threshold. ### Claim 6 (depends on 5) The method of claim 5, wherein the training comprises: increasing the confidence threshold as training progresses. ### Claim 7 (depends on 6) The method of claim 6, wherein increasing the confidence threshold comprises increasing the confidence threshold after each training step. ### Claim 8 (depends on 1) The method of claim 1, wherein the machine learning model is trained to map model inputs comprising visual data to model outputs, thereby to perform a computer vision task. ### Claim 9 (depends on 1) The method of claim 1, wherein the machine learning task is image classification, the model inputs are images and the model outputs for a given image are probabilities for each of a set of object categories, each probability representing an estimated likelihood that the image contains an object belonging to the category. ### Claim 10 (depends on 1) The method of claim 1, wherein the machine learning model is trained to map inputs comprising audio data representing a spoken utterance to an output in order to perform a speech classification task. ### Claim 11 (depends on 1) The method of claim 1, wherein a model input comprises electronic health record data for a patient and the corresponding model output comprises a probability distribution over possible diagnoses for the patient. ### Claim 12 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one more computers to perform -operations comprising: receiving training data for training a machine learning model to map model inputs to model outputs in order to perform a particular machine learning task, the training data comprising: a plurality of unlabeled training inputs; and a plurality of labeled training inputs and, for each labeled training input, a ground truth output that should be generated by the machine learning model by performing the particular machine learning task on the labeled training input; generating augmented training data, comprising generating, for each of the plurality of unlabeled training inputs, a respective augmented training input by applying a data augmentation technique to the unlabeled training input; training the machine learning model on the augmented training data, comprising: training the machine learning model on the unlabeled training inputs and the augmented training inputs to optimize an unsupervised objective that measures a difference between (i) a model output generated by the machine learning model for a given unlabeled training input and (ii) a model output generated by the machine learning model for the augmented training input generated from the unlabeled training input, and training the machine learning model on the labeled training inputs to optimize a supervised objective that measures a difference between (i) a model output generated by the machine learning model for a given labeled training input and (ii) the ground truth output for the given labeled training input. ### Claim 13 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations comprising: receiving training data for training a machine learning model to map model inputs to model outputs in order to perform a particular machine learning task, the training data comprising: a plurality of unlabeled training inputs; and a plurality of labeled training inputs and, for each labeled training input, a ground truth output that should be generated by the machine learning model by performing the particular machine learning task on the labeled training input; generating augmented training data, comprising generating, for each of the plurality of unlabeled training inputs, a respective augmented training input by applying a data augmentation technique to the unlabeled training input; training the machine learning model on the augmented training data, comprising: training the machine learning model on the unlabeled training inputs and the augmented training inputs to optimize an unsupervised objective that measures a difference between (i) a model output generated by the machine learning model for a given unlabeled training input and (ii) a model output generated by the machine learning model for the augmented training input generated from the unlabeled training input, and training the machine learning model on the labeled training inputs to optimize a supervised objective that measures a difference between (i) a model output generated by the machine learning model for a given labeled training input and (ii) the ground truth output for the given labeled training input. ### Claim 14 (depends on 13) The system of claim 13, wherein the model outputs are probability distributions and wherein the unsupervised objective is based on a K-L divergence between (i) the model output generated by the machine learning model for the given unlabeled training input and (ii) the model output generated by the machine learning model for the augmented training input generated from the unlabeled training input. ### Claim 15 (depends on 13) The system of claim 13, wherein the model outputs are probability distributions and wherein the supervised objective is based on a negative log likelihood of the given ground truth output according to the probability distribution generated for the given labeled training input. ### Claim 16 (depends on 13) The system of claim 13, wherein the labeled and unlabeled training inputs in the augmented training data have been augmented by applying a different data augmentation technique from the data augmentation technique used to generate the augmented unlabeled training inputs. ### Claim 17 (depends on 13) The system of claim 13, wherein the model outputs are probability distributions, wherein the supervised objective is: based on a negative log likelihood of the given ground truth output according to the probability distribution generated for the given labeled training input when the probability assigned to the given ground truth output by the probability distribution is less than a confidence threshold, and equal to zero when the probability assigned to the given ground truth output by the probability distribution is equal to or greater than the confidence threshold. ### Claim 18 (depends on 17) The system of claim 17, wherein the training comprises: increasing the confidence threshold as training progresses. ### Claim 19 (depends on 18) The system of claim 18, wherein increasing the confidence threshold comprises increasing the confidence threshold after each training step. ### Claim 20 (depends on 13) The system of claim 13, wherein the machine learning model is trained to map model inputs comprising visual data to model outputs, thereby to perform a computer vision task.