Reinforcement learning using advantage estimates

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for controlling a robot interacting with a real-world environment, the operations comprising:

• receiving an observation characterizing a current state of the environment;
• and
• processing the observation using a policy subnetwork to generate an output action from a set of actions that lie on a continuous domain, wherein:
• the output action comprises a plurality of control inputs for the robot;
• the policy subnetwork has been trained jointly with a value subnetwork on a set of training experience tuples using an error function,
• the value subnetwork is configured to process an input observation characterizing an input state to generate as output a value estimate for the input state, the value estimate being an estimate of an expected return resulting from the environment being in the input state irrespective of which action is performed in response to the input observation,
• each training experience tuple identifies i) a training observation characterizing a training state of a training environment, ii) a training action performed by an agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment, and
• the error function for the joint training of the value subnetwork and the policy subnetwork measures, for each training experience tuple, an error between i) a Q value for the training action identified in the training observation that is generated using an advantage estimate for the training action that is based on a distance between a) an output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation and ii) a target Q value that is generated based on c) the reward identified in the training experience tuple and d) a value estimate generated by processing the subsequent observation identified in the training experience tuple using the value subnetwork;
• and
• causing the robot to perform the output action by providing the plurality of control inputs to the robot.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for controlling a robot interacting with a real-world environment, the operations comprising:

• receiving an observation characterizing a current state of the environment;
• and
• processing the observation using a policy subnetwork to generate an output action from a set of actions that lie on a continuous domain, wherein:
• the output action comprises a plurality of control inputs for the robot;
• the policy subnetwork has been trained jointly with a value subnetwork on a set of training experience tuples using an error function,
• the value subnetwork is configured to process an input observation characterizing an input state to generate as output a value estimate for the input state, the value estimate being an estimate of an expected return resulting from the environment being in the input state irrespective of which action is performed in response to the input observation,
• each training experience tuple identifies i) a training observation characterizing a training state of a training environment, ii) a training action performed by an agent in response to the training observation, iii) a reward received as a result of the agent performing the training action in response to the training observation, and iv) a subsequent observation characterizing a subsequent state of the environment, and
• the error function for the joint training of the value subnetwork and the policy subnetwork measures, for each training experience tuple, an error between i) a Q value for the training action identified in the training observation that is generated using an advantage estimate for the training action that is based on a distance between a) an output action generated by processing the training observation using the policy subnetwork and b) the training action identified in the training observation and ii) a target Q value that is generated based on c) the reward identified in the training experience tuple and d) a value estimate generated by processing the subsequent observation identified in the training experience tuple using the value subnetwork;
• and
• causing the robot to perform the output action by providing the plurality of control inputs to the robot.