Transfer learning between different computer vision tasks

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network comprising a plurality of neural network layers to perform a downstream computer vision task, each of the plurality of neural network layers having a respective plurality of layer parameters, and the neural network being configured to receive a network input comprising one or more images to and to process the network input to generate a network output for the downstream computer vision task, the method comprising:

• pre-training an initial neural network comprising a first subset of the plurality of neural network layers on training data for an initial computer vision task through supervised learning to determine first values of the respective layer parameters of the first subset of neural network layers, wherein the first subset of the plurality of neural network layers comprises:
• a first convolutional neural network layer with weight standardization (WS) followed by a group normalization (GN) layer;
• and
• training the neural network on training data for the downstream computer vision task through supervised learning to determine trained values of the respective layer parameters of the first subset of neural network layers from the first values of the respective layer parameters of the first subset of neural network layers.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network comprising a plurality of neural network layers to perform a downstream computer vision task, each of the plurality of neural network layers having a respective plurality of layer parameters, and the neural network being configured to receive a network input comprising one or more images to and to process the network input to generate a network output for the downstream computer vision task, the method comprising:

• pre-training an initial neural network comprising a first subset of the plurality of neural network layers on training data for an initial computer vision task through supervised learning to determine first values of the respective layer parameters of the first subset of neural network layers, wherein the first subset of the plurality of neural network layers comprises:
• a first convolutional neural network layer with weight standardization (WS) followed by a group normalization (GN) layer;
• and
• training the neural network on training data for the downstream computer vision task through supervised learning to determine trained values of the respective layer parameters of the first subset of neural network layers from the first values of the respective layer parameters of the first subset of neural network layers.