# US 2019/0314985 A1 · Mirror loss neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0314985 A1 - Document ID: US20190314985A1 - Status: Published application - Kind: A1 - Application number: 16/468,987 - Filed: 2018-03-19 - Published: 2019-10-17 - Earliest claimed: 2017-03-17 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Pierre Sermanet - Web page: https://patentgrep.com/patent/US20190314985A1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1697: {Vision controlled systems} - G05B 13/027: {using neural networks only} - G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/82: using neural networks - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - H04N 7/181: {for receiving images from a plurality of remote sources} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This description relates to a neural network that has multiple network parameters and is configured to receive an input observation characterizing a state of an environment and to process the input observation to generate a numeric embedding of the state of the environment. The neural network can be used to control a robotic agent. The network can be trained using a method comprising: obtaining a first observation captured by a first modality; obtaining a second observation that is co-occurring with the first observation and that is captured by a second, different modality; obtaining a third observation captured by the first modality that is not co-occurring with the first observation; determining a gradient of a triplet loss that uses the first observation, the second observation, and the third observation; and updating current values of the network parameters using the gradient of the triplet loss. ## Claims ### Claim 1 A method of training a neural network having a plurality of network parameters, wherein the neural network is configured to receive an input observation characterizing a state of an environment and to process the input observation to generate a numeric embedding of the state of the environment, the method comprising: obtaining a first observation captured by a first modality; obtaining a second observation that is co-occurring with the first observation and that is captured by a second, different modality; obtaining a third observation captured by the first modality that is not co-occurring with the first observation; determining a gradient of a triplet loss that uses the first observation as an anchor example, the second observation as a positive example, and the third observation as a negative example; and updating current values of the network parameters using the gradient of the triplet loss. ### Claim 2 (depends on 1) The method of claim 1, wherein the observations are images, wherein the first modality is a camera at a first viewpoint, and wherein the second modality is a camera at a second, different viewpoint. ### Claim 3 (depends on 1) The method of claim 1, wherein the third observation is within a temporal neighborhood of the first observation. ### Claim 4 (depends on 3) The method of claim 3, wherein obtaining the third observation comprises: selecting the third observation randomly from the observations captured by the first modality that are within the temporal neighborhood of the first observation. ### Claim 5 (depends on 3) The method of claim 3, wherein obtaining the third observation comprises: selecting an observation that is a hard negative relative to the first observation from the observations captured by the first modality that are within the temporal neighborhood of the first observation. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the gradient of the triplet loss comprises: processing the first observation using the neural network in accordance with the current values of the network parameters to generate a first embedding; processing the second observation using the neural network in accordance with the current values of the network parameters to generate a second embedding; processing the third observation using the neural network in accordance with the current values of the network parameters to generate a third embedding; and determining the triplet loss from (i) a first distance between the first embedding and the second embedding and (ii) a second distance between the first embedding and the third embedding. ### Claim 7 (depends on 1) The method of claim 1, wherein the neural network is further configured to process the numeric embedding to generate a control output defining a control input to an agent interacting with the environment. ### Claim 8 (depends on 7) The method of claim 7, wherein training the neural network further comprises training the neural network to generate control outputs that cause the agent to imitate another agent interacting with the environment. ### Claim 9 (depends on 1) The method of claim 1, wherein the first modality is a camera that captures a self-view of the agent interacting with the environment, and wherein training the neural network further comprises training the neural network to generate control outputs that position joints of the agent in a same position as in received captured self-views. ### Claim 10 (depends on 7) The method of claim 7, wherein the agent is a robotic agent. ### Claim 11 (depends on 1) The method of claim 1, wherein the environment includes an agent performing a specified task. ### Claim 12 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network having a plurality of network parameters, wherein the neural network is configured to receive an input observation characterizing a state of an environment and to process the input observation to generate a numeric embedding of the state of the environment, the operations comprising: obtaining a first observation captured by a first modality; obtaining a second observation that is co-occurring with the first observation and that is captured by a second, different modality; obtaining a third observation captured by the first modality that is not co-occurring with the first observation; determining a gradient of a triplet loss that uses the first observation as an anchor example, the second observation as a positive example, and the third observation as a negative example; and updating current values of the network parameters using the gradient of the triplet loss. ### Claim 13 One or more non-transitory computer storage media storing instructions that when executed by one or more computers, cause the one or more computers to perform operations for training a neural network having a plurality of network parameters, wherein the neural network is configured to receive an input observation characterizing a state of an environment and to process the input observation to generate a numeric embedding of the state of the environment, the operations comprising: obtaining a first observation captured by a first modality; obtaining a second observation that is co-occurring with the first observation and that is captured by a second, different modality; obtaining a third observation captured by the first modality that is not co-occurring with the first observation; determining a gradient of a triplet loss that uses the first observation as an anchor example, the second observation as a positive example, and the third observation as a negative example; and updating current values of the network parameters using the gradient of the triplet loss. ### Claim at position 14 -17. (canceled) ### Claim 18 (depends on 13) The one or more non-transitory computer storage media of claim 13, wherein the observations are images, wherein the first modality is a camera at a first viewpoint, and wherein the second modality is a camera at a second, different viewpoint. ### Claim 19 (depends on 13) The one or more non-transitory computer storage media of claim 13, wherein the third observation is within a temporal neighborhood of the first observation. ### Claim 20 (depends on 19) The one or more non-transitory computer storage media of claim 19, wherein obtaining the third observation comprises: selecting the third observation randomly from the observations captured by the first modality that are within the temporal neighborhood of the first observation. ### Claim 21 (depends on 19) The one or more non-transitory computer storage media of claim 19, wherein obtaining the third observation comprises: selecting an observation that is a hard negative relative to the first observation from the observations captured by the first modality that are within the temporal neighborhood of the first observation. ### Claim 22 (depends on 13) The one or more non-transitory computer storage media of claim 13, wherein determining the gradient of the triplet loss comprises: processing the first observation using the neural network in accordance with the current values of the network parameters to generate a first embedding; processing the second observation using the neural network in accordance with the current values of the network parameters to generate a second embedding; processing the third observation using the neural network in accordance with the current values of the network parameters to generate a third embedding; and determining the triplet loss from (i) a first distance between the first embedding and the second embedding and (ii) a second distance between the first embedding and the third embedding. ### Claim 23 (depends on 13) The one or more non-transitory computer storage media of claim 13, wherein the neural network is further configured to process the numeric embedding to generate a control output defining a control input to an agent interacting with the environment. ### Claim 24 (depends on 23) The one or more non-transitory computer storage media of claim 23, wherein training the neural network further comprises training the neural network to generate control outputs that cause the agent to imitate another agent interacting with the environment.