Continuous control with deep reinforcement learning

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• receiving a current observation characterizing a current state of an environment;
• processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performed by an agent in response to the input observation;
• and
• causing the agent to perform the action defined by the point in the continuous action space, wherein the actor neural network has been trained jointly with a critic neural network that is configured to receive a critic input specifying a training observation and a training action and to process the critic input to generate a neural network output that is an estimate of time-discounted total future rewards that will be received in response to the agent performing the training action in response to the training observation, and wherein jointly training the critic neural network and the actor neural network comprises:
• obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of an environment, a training action from a continuous space of actions performed by an agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment;
• and
• updating current values of parameters of the actor neural network using the minibatch of experience tuples, wherein the updating comprises:
• for each experience tuple in the minibatch:
• processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network;
• processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network has a same neural network architecture as the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;
• processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network has a same neural network architecture as the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network;
• and
• determining a target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple,
• updating the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch;
• and
• updating current values of the parameters of the actor neural network using the critic neural network.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving a current observation characterizing a current state of an environment;
• processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performed by an agent in response to the input observation;
• and
• causing the agent to perform the action defined by the point in the continuous action space, wherein the actor neural network has been trained jointly with a critic neural network that is configured to receive a critic input specifying a training observation and a training action and to process the critic input to generate a neural network output that is an estimate of time-discounted total future rewards that will be received in response to the agent performing the training action in response to the training observation, and wherein jointly training the critic neural network and the actor neural network comprises:
• obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of an environment, a training action from a continuous space of actions performed by an agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment;
• and
• updating current values of parameters of the actor neural network using the minibatch of experience tuples, wherein the updating comprises:
• for each experience tuple in the minibatch:
• processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network;
• processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network has a same neural network architecture as the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;
• processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network has a same neural network architecture as the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network;
• and
• determining a target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple,
• updating the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch;
• and
• updating current values of the parameters of the actor neural network using the critic neural network.