Semi-supervised training of neural networks

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network that has a plurality of network parameters and that is configured to receive an input data item comprising an image and to process the input data item to generate an embedding of the input data item in accordance with the network parameters, wherein the neural network is further configured to process the embedding of the data item to generate a classification output for the data item that includes a respective score for each of a plurality of categories, the operations comprising:

• obtaining a labeled batch of labeled training items, wherein each labeled training item in the labeled batch is associated with a respective label that identifies a category to which the labeled training item belongs:
• processing the labeled training items in the labeled batch using the neural network and in accordance with current values of the network parameters to generate respective embeddings for each of the labeled training items:
• obtaining an unlabeled batch of unlabeled training items;
• processing the unlabeled training items in the unlabeled batch using the neural network and in accordance with the current values of the network parameters to generate respective embeddings for each of the unlabeled training items;
• determining a plurality of similarity values, each similarity value measuring a similarity between the embedding for a respective labeled training item and the embedding for a respective unlabeled training item;
• determining, from the similarity values, a respective roundtrip path probability for each of a plurality of roundtrip paths, wherein each roundtrip path starts from the embedding for a respective starting labeled training item, goes to the embedding for a respective unlabeled training item, and returns to the embedding for a respective ending labeled training item, wherein determining the respective roundtrip path probability for each of the plurality of roundtrip paths comprises:
• determining a forward path probability for a forward path from the embedding for the starting labeled training item for the roundtrip path to the embedding for the unlabeled training item for the roundtrip path based on the similarity value between the embedding for the starting labeled training item for the roundtrip path to the embedding for the unlabeled training item for the roundtrip path:
• determining a backward path probability for a backward path from the embedding for the unlabeled training item for the roundtrip path to the embedding for the ending labeled training item for the roundtrip path based on the similarity value between the embedding for the unlabeled training item for the roundtrip path and the embedding for the ending labeled training item for the roundtrip path;
• and
• determining the roundtrip path probability from the product of the forward path probability and the backward path probability;
• and
• performing an iteration of a neural network training procedure to determine a first value update to the current values of the network parameters that decreases roundtrip path probabilities for incorrect roundtrip paths, wherein an incorrect roundtrip path is a roundtrip path for which the starting labeled training item and the ending labeled training item have different labels.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network that has a plurality of network parameters and that is configured to receive an input data item comprising an image and to process the input data item to generate an embedding of the input data item in accordance with the network parameters, wherein the neural network is further configured to process the embedding of the data item to generate a classification output for the data item that includes a respective score for each of a plurality of categories, the operations comprising:

• obtaining a labeled batch of labeled training items, wherein each labeled training item in the labeled batch is associated with a respective label that identifies a category to which the labeled training item belongs;
• processing the labeled training items in the labeled batch using the neural network and in accordance with current values of the network parameters to generate respective embeddings for each of the labeled training items;
• obtaining an unlabeled batch of unlabeled training items;
• processing the unlabeled training items in the unlabeled batch using the neural network and in accordance with the current values of the network parameters to generate respective embeddings for each of the unlabeled training items;
• determining a plurality of similarity values, each similarity value measuring a similarity between the embedding for a respective labeled training item and the embedding for a respective unlabeled training item;
• determining, from the similarity values, a respective roundtrip path probability for each of a plurality of roundtrip paths, wherein each roundtrip path starts from the embedding for a respective starting labeled training item, goes to the embedding for a respective unlabeled training item, and returns to the embedding for a respective ending labeled training item, wherein determining the respective roundtrip path probability for each of the plurality of roundtrip paths comprises:
• determining a forward path probability for a forward path from the embedding for the starting labeled training item for the roundtrip path to the embedding for the unlabeled training item for the roundtrip path based on the similarity value between the embedding for the starting labeled training item for the roundtrip path to the embedding for the unlabeled training item for the roundtrip path;
• determining a backward path probability for a backward path from the embedding for the unlabeled training item for the roundtrip path to the embedding for the ending labeled training item for the roundtrip path based on the similarity value between the embedding for the unlabeled training item for the roundtrip path and the embedding for the ending labeled training item for the roundtrip path;
• and
• determining the roundtrip path probability from the product of the forward path probability and the backward path probability;
• and
• performing an iteration of a neural network training procedure to determine a first value update to the current values of the network parameters that decreases roundtrip path probabilities for incorrect roundtrip paths, wherein an incorrect roundtrip path is a roundtrip path for which the starting labeled training item and the ending labeled training item have different labels.