Control policies for collective robot learning

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a system for training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,

• wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
• wherein the system comprises:
• a plurality of local workers, wherein each local worker selects actions to be performed by a different robotic agent interacting with a different instance of the real-world environment, wherein each local worker maintains a respective local policy controller having a different model architecture than the global policy neural network, and wherein each local worker is configured to repeatedly perform the following operations asynchronously from each other local worker:
• initializing an instance of the robotic task,
• generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task,
• optimizing a local policy controller of the local worker on the trajectory using a first machine learning training technique,
• generating an optimized trajectory using the optimized local policy controller of the local worker, and
• storing the optimized trajectory in a replay memory associated with the local worker;
• and
• a plurality of global workers that operate asynchronously from the plurality of local workers and in parallel with the plurality of local workers, wherein each global worker maintains a respective replica of the global policy neural network and trains the replica of the global policy neural network using optimized trajectories sampled from one or more replay memories of the local workers, and wherein each global worker is configured to repeatedly perform the following operations asynchronously from the operations performed by the plurality of local workers and in parallel with the plurality of operations performed by the local workers:
• sampling an optimized trajectory, from one of one or more replay memories associated with the global worker, that is generated by a local worker of the plurality of local workers using a local policy controller that is optimized using the first machine learning training technique and that has a different model architecture than the global policy neural network;
• and
• training the replica of the global policy neural network maintained by the global worker on the sampled optimized trajectory using a second machine learning training technique which is different than the first machine learning training technique used to train the local policy controllers of the local workers to determine delta values that define an update to the parameters of the global policy neural network.

A method for training a global policy neural network used to select actions to be performed by a robotic agent interacting with a real-world environment to perform a robotic task,

• wherein the global policy neural network is configured to receive as input an observation characterizing a state of the environment and to generate a global policy output in accordance with current values of a plurality of parameters, and
• wherein the method comprises:
• repeatedly performing first operations by each of a plurality of local workers asynchronously from each other worker, wherein each local worker selects actions to be performed by a different robotic agent interacting with a different instance of the real-world environment, wherein each local worker maintains a respective local policy controller having a different model architecture than the global policy neural network, and wherein the first operations comprise:
• initializing an instance of the robotic task,
• generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task,
• optimizing a local policy controller of the local worker on the trajectory using a first machine learning training technique,
• generating an optimized trajectory using the optimized local policy controller of the local worker, and
• storing the optimized trajectory in a replay memory associated with the local worker;
• and
• repeatedly performing second operations by each of a plurality of global workers that operate asynchronously from the plurality of local workers and in parallel with the plurality of local workers, wherein each global worker maintains a respective replica of the global policy neural network and trains the replica of the global policy neural network using optimized trajectories sampled from one or more replay memories of the local workers, and wherein the second operations comprise:
• sampling an optimized trajectory, from one of one or more replay memories associated with the global worker, that is generated by a local worker of the plurality of local workers using a local policy controller that is optimized using the first machine learning training technique and that has a different model architecture than the global policy neural network;
• and
• training the replica of the global policy neural network maintained by the global worker on the sampled optimized trajectory using a second machine learning training technique which is different than the first machine learning training technique used to train the local policy controllers of the local workers to determine delta values that define an update to the parameters of the global policy neural network.