Training neural networks using consistency measures

A system comprising a respective training system for each of a plurality of neural networks, wherein each neural network is configured to perform a respective different machine learning task, and wherein the training system for each first neural network of the plurality of neural networks corresponding to a first machine learning task comprises:

• one or memories configured to store a copy of a mediator training data set comprising a plurality of unlabeled training examples, wherein the mediator training data set is shared by each training system;
• a respective forward-pass engine corresponding to each second neural network of the plurality of neural networks that is not the first neural network, wherein the forward pass engine for each second neural network is configured to perform operations comprising:
• obtaining parameter values for the respective second neural network;
• receive a particular training example in the mediator training data set;
• and
• processing the particular training example using the respective second neural network to generate a second output for the respective second machine learning task that is different from the first machine learning task;
• and
• a training engine configured to perform operations comprising:
• processing a particular training example in the mediator training data set using the first neural network to generate a first output for the first machine learning task;
• obtaining a respective second output from each of the one or more forward-pass engines, wherein each second output corresponds to the machine learning task of the respective second neural network that is different from the first machine learning task;
• determining, from each second output, a respective consistency target output for the first machine learning task that would be consistent with a relationship between outputs for the first machine learning task and outputs for the machine learning task corresponding to the second output;
• determining, from each second output, an error between the first output and the respective consistency target output;
• and
• generating a parameter update for the first neural network from the error.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform a method for training a first neural network having a plurality of first network parameters to perform a first machine learning task, the method comprising:

• obtaining a mediator training data set comprising a plurality of unlabeled training examples;
• processing a particular training example in the mediator training data set using the first neural network to generate a first output for the first machine learning task;
• processing the particular training example in the mediator training data set using each of one or more second neural networks, wherein each second neural network is configured to process the particular training example to generate a second output for a respective second machine learning task that is different from the first machine learning task;
• determining, for each second machine learning task and from the second output of the corresponding second neural network, a consistency target output for the first machine learning task that would be consistent with a relationship between outputs for the first machine learning task and outputs for the second machine learning task;
• determining, for each second machine learning task, an error between the first output and the consistency target output corresponding to the second machine learning task;
• and
• generating a parameter update for the first neural network from the determined errors.