Training neural networks using a clustering loss

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations, the operations comprising:

• training a neural network to perform an image processing task by receiving and processing an input image, wherein the neural network has a plurality of network parameters and is configured to generate an embedding of the input image in accordance with the network parameters, wherein training the neural network comprises, at each of plurality of training iterations:
• obtaining a batch of training images for the training iteration and a ground truth assignment of the training images in the batch into a plurality of clusters;
• processing the training images in the batch using the neural network and in accordance with current values of the network parameters to generate respective embeddings for each of the training images;
• determining an oracle clustering score for the ground truth assignment based on the respective embeddings;
• wherein the oracle clustering score measures a quality of the ground truth assignment based on the respective embeddings of each of the training images in the batch;
• determining, from a plurality of clustering assignments, a possible clustering assignment other than the ground truth assignment that has a highest augmented clustering score, wherein:
• each clustering assignment assigns the training images in the batch into a respective plurality of clusters that are each represented by a respective medoid,
• the augmented clustering score is a clustering score for the possible clustering assignment plus a structured margin between the possible clustering assignment and the ground truth assignment,
• the clustering score for a clustering assignment measuring for each embedding of the training inputs, how close the embedding is to a closest medoid to the embedding, and
• the structured margin measures a quality of the possible clustering assignment relative to the ground truth assignment;
• and
• training the neural network using the possible clustering assignment other than the ground truth assignment that has a highest augmented clustering score to update the current values of the network parameter;
• after training the neural network, performing, using the neural network, the image processing task on a new input image, wherein the image processing task comprises at least one of image classification, feature based retrieval, image clustering, near duplicate image detection, feature matching, domain adaptation, or video based weakly supervised learning.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations, the operations comprising:

• training a neural network to perform an image processing task by receiving and processing an input image, wherein the neural network has a plurality of network parameters and is configured to generate an embedding of the input image in accordance with the network parameters, wherein training the neural network comprises:
• obtaining a batch of training images and a ground truth assignment of the training images in the batch into a plurality of clusters;
• processing the training images in the batch using the neural network and in accordance with current values of the network parameters to generate respective embeddings for each of the training images;
• determining an oracle clustering score for the ground truth assignment based on the respective embeddings;
• and
• wherein the oracle clustering score measures a quality of the ground truth assignment based on the respective embeddings of each of the training images in the batch;
• determining, from a plurality of clustering assignments, a possible clustering assignment other than the ground truth assignment that has a highest augmented clustering score, wherein:
• each clustering assignment assigns the training images in the batch into a respective plurality of clusters that are each represented by a respective medoid,
• the augmented clustering score is a clustering score for the possible clustering assignment plus a structured margin between the possible clustering assignment and the ground truth assignment,
• the clustering score for a clustering assignment measuring for each embedding of the training inputs, how close the embedding is to a closest medoid to the embedding, and
• the structured margin measures a quality of the possible clustering assignment relative to the ground truth assignment;
• and
• training the neural network using the possible clustering assignment other than the ground truth assignment that has a highest augmented clustering score to update the current values of the network parameter;
• after training the neural network, performing, using the neural network, the image processing task on a new input image, wherein the image processing task comprises at least one of image classification, feature based retrieval, image clustering, near duplicate image detection, feature matching, domain adaptation, or video based weakly supervised learning.