Robust reinforcement learning for continuous control with model misspecification

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a policy neural network having a plurality of policy network parameters,

• wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment,
• wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and
• wherein the operations comprise:
• sampling a mini-batch comprising one or more observation—action—reward tuples generated as a result of interactions of a first agent with a first environment;
• determining an update to current values of the Q network parameters, comprising:
• for each tuple and for each possible perturbation of the environment in a set of a plurality of possible perturbations of the environment:
• causing the agent to perform the action in the tuple when the state of the environment represented by the observation in the tuple has been perturbed by applying the possible perturbation to the state of the environment represented by the observation;
• and
• in response, obtaining a next observation characterizing a perturbed next state;
• and
• minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation—action—reward tuples and measures, for each tuple, an error between (i) a target Q value determined based on the reward in the tuple and an infimum or average of entropy-regularized Q values for the perturbed next states for the plurality of possible perturbations and (ii) a Q value for the observation-action pair in the tuple;
• and
• determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation-action-reward tuples.

A system comprising one or more computers and one or more storage devices storing instruction that when executed by one or more computers cause the one or more computers to perform operations fort raining a policy neural network having a plurality of policy network parameters,

• wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment,
• wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and
• wherein the operations comprise:
• sampling a mini-batch comprising one or more observation—action—reward tuples generated as a result of interactions of a first agent with a first environment;
• determining an update to current values of the Q network parameters, comprising:
• for each tuple and for each possible perturbation of the environment in a set of a plurality of possible perturbations of the environment:
• causing the agent to perform the action in the tuple when the state of the environment represented by the observation in the tuple has been perturbed by applying the possible perturbation to the state of the environment represented by the observation;
• and
• in response, obtaining a next observation characterizing a perturbed next state;
• and
• minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation—action—reward tuples and measures, for each tuple, an error between (i) a target Q value determined based on the reward in the tuple and an infimum or average of entropy-regularized Q values for the perturbed next states for the plurality of possible perturbations and (ii) a Q value for the observation-action pair in the tuple;
• and
• determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation—action—reward tuples.