Neural networks for selecting actions to be performed by a robotic agent

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;
• processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and
• wherein the sequence of DNN s comprises:
• a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein:
• the simulation-trained DNN comprises a first plurality of indexed layers, and
• the simulation-trained DNN, implemented by the one or more computers, is configured to receive the observation and process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers;
• and
• a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed,
• wherein
• the first robot-trained DNN comprises a second plurality of indexed layers,
• the first robot-trained DNN, implemented by the one or more computers, is configured to receive the observation and to process the observation through each layer in the second plurality of indexed layers to generate the policy output, and
• one or more of the layers in the second plurality of indexed layers, implemented by the one or more computers, are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation-trained DNN, wherein a preceding layer is a layer whose index is one less than the index of the layer;
• and
• selecting an action to be performed by the robotic agent in response to the observation using the policy output.

A system comprising:

• one or more computers;
• and
• one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;
• processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and
• wherein the sequence of DNN s comprises:
• a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein:
• the simulation-trained DNN comprises a first plurality of indexed layers,
• and
• the simulation-trained DNN, implemented by the one or more computers, is configured to receive the observation and process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers;
• and
• a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed,
• wherein
• the first robot-trained DNN comprises a second plurality of indexed layers,
• the first robot-trained DNN, implemented by the one or more computers, is configured to receive the observation and to process the observation through each layer in the second plurality of indexed layers to generate the policy output, and
• one or more of the layers in the second plurality of indexed layers, implemented by the one or more computers, are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation-trained DNN, wherein a preceding layer is a layer whose index is one less than the index of the layer;
• and
• selecting an action to be performed by the robotic agent in response to the observation using the policy output.

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system, implemented by the one or more computers, is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,

• wherein the sequence of DNN s comprises:
• a simulation-trained DNN, wherein the simulation-trained DNN, implemented by the one or more computers, comprises a first plurality of indexed layers and is configured to receive the observation and process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers;
• and
• a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein
• the first robot-trained DNN comprises a second plurality of indexed layers,
• the first robot-trained DNN, implemented by the one or more computers, is configured to receive the observation and to process the observation through each layer in the second plurality of indexed layers to generate the policy output, and
• one or more of the layers in the second plurality of indexed layers, implemented by the one or more computers, are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation-trained DNN, wherein a preceding layer is a layer whose index is one less than the index of the layer,
• wherein the operations comprise:
• training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed.

A system comprising:

• one or more computers;
• and
• one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system, implemented by the one or more computers, is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,
• wherein the sequence of DNN s comprises:
• a simulation-trained DNN, wherein the simulation-trained DNN, implemented by the one or more computers, comprises a first plurality of indexed layers and is configured to receive the observation and process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers;
• and
• a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein
• the first robot-trained DNN comprises a second plurality of indexed layers,
• the first robot-trained DNN, implemented by the one or more computers, is configured to receive the observation and to process the observation through each layer in the second plurality of indexed layers to generate the policy output, and
• one or more of the layers in the second plurality of indexed layers, implemented by the one or more computers, are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation-trained DNN, wherein a preceding layer is a layer whose index is one less than the index of the layer,
• wherein the operations comprise:
• training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed.