Neural networks for selecting actions to be performed by a robotic agent

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;
• processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and
• wherein the sequence of DNNs comprises:
• a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, and
• a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output;
• and
• selecting an action to be performed by the robotic agent in response to the observation using the policy output,
• wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

A system comprising:

• one or more computers;
• and
• one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;
• processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and
• wherein the sequence of DNNs comprises:
• a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, and
• a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output;
• and
• selecting an action to be performed by the robotic agent in response to the observation using the policy output,
• wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system, implemented by the one or more computers, is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,

• wherein the sequence of DNNs comprises:
• a simulation-trained DNN, the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, wherein the simulation-trained DNN is trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task;
• and
• a first robot-trained DNN that is trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output;
• and
• wherein the operations comprise:
• training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed,
• wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

A system comprising:

• one or more computers;
• and
• one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system, implemented by the one or more computers, is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,
• wherein the sequence of DNNs comprises:
• a simulation-trained DNN, the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, wherein the simulation-trained DNN is trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task;
• and
• a first robot-trained DNN that is trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output;
• and
• wherein the operations comprise:
• training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed,
• wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.