Mirror loss neural networks

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network having a plurality of network parameters, wherein the neural network is configured to receive an input observation characterizing a state of an environment and to process the input observation to generate a numeric embedding of the state of the environment, the operations comprising:

• obtaining a first observation captured by a first modality;
• obtaining a second observation that is co-occurring with the first observation and that is captured by a second, different modality;
• obtaining a third observation captured by the first modality that is not co-occurring with the first observation;
• determining a gradient of a triplet loss that uses the first observation as an anchor example, the second observation as a positive example, and the third observation as a negative example;
• and
• updating current values of the network parameters using the gradient of the triplet loss,
• wherein the observations are images related to a same task, wherein the first modality is a camera at a first viewpoint, and wherein the second modality is a camera at a second, different viewpoint.

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network having a plurality of network parameters, wherein the neural network is configured to receive an input observation characterizing a state of an environment and to process the input observation to generate a numeric embedding of the state of the environment, the operations comprising:

• obtaining a first observation captured by a first modality;
• obtaining a second observation that is co-occurring with the first observation and that is captured by a second, different modality;
• obtaining a third observation captured by the first modality that is not co-occurring with the first observation;
• determining a gradient of a triplet loss that uses the first observation as an anchor example, the second observation as a positive example, and the third observation as a negative example;
• and
• updating current values of the network parameters using the gradient of the triplet loss,
• wherein the observations are images related to a same task, wherein the first modality is a camera at a first viewpoint, and wherein the second modality is a camera at a second, different viewpoint.