Systems and methods for contrastive learning of visual representations

A computing system for performing semi-supervised contrastive learning of visual representations, the computing system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store:
• an image classification model comprising a base encoder neural network, one or more projection head neural network layers, and a classification head,
• wherein the base encoder neural network and the one or more projection head neural network layers have been pretrained using contrastive learning based on a set of one or more unlabeled visual data, and
• wherein the one or more projection head neural network layers comprise some but not all of a plurality of projection head neural network layers from a projection head neural network employed during said contrastive learning;
• and
• instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising:
• performing fine-tuning of the image classification model using a set of one or more labeled visual data;
• and
• after performing the fine-tuning of the image classification model, performing distillation training using the one or more projection head neural network layers pretrained using contrastive learning, wherein the distillation training distills the image classification model to a student model comprising a relatively smaller number of parameters relative to the image classification model.

A computer-implemented method for performing semi-supervised contrastive learning, the method comprising:

• performing contrastive learning based on a set of one or more unlabeled training data examples;
• generating an image classification model based on a base encoder neural network used in performing the contrastive learning and based on some but not all of a plurality of layers in a projection head neural network used in performing the contrastive learning;
• performing fine-tuning of the image classification model based on a set of one or more labeled training data;
• and
• after performing the fine-tuning of the image classification model, performing distillation training using the set of unlabeled training data examples, the distillation training distilling the image classification model to a student model comprising a relatively smaller number of parameters relative to the image classification model.