Control policies for robotic agents

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,

• wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
• wherein the operations comprise:
• initializing a plurality of instances of the robotic task, comprising randomly selecting a respective initial state of the environment for each instance;
• for each instance:
• generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
• optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
• generating training data for the global policy neural network using the local policy controllers;
• and
• training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,

• wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
• wherein the operations comprise:
• initializing a plurality of instances of the robotic task, comprising randomly selecting a respective initial state of the environment for each instance;
• for each instance:
• generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
• optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
• generating training data for the global policy neural network using the local policy controllers;
• and
• training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.

A method of training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,

• wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters,
• wherein the global policy output defines a distribution over a plurality of possible actions, and
• wherein the method comprises:
• initializing a plurality of instances of the robotic task;
• for each instance:
• generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and
• optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance;
• generating training data for the global policy neural network using the local policy controllers;
• and
• training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.