Deep reinforcement learning with fast updating recurrent neural networks and slow updating recurrent neural networks

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent interacting with an environment, the operations comprising, at each of a plurality of time steps:

• receiving an observation characterizing a current state of the environment at the time step;
• and
• selecting an action to be performed by the agent in response to the observation based on (i) the observation, (ii) a first prior distribution over possible latent variables generated from a hidden state of a slow updating recurrent neural network that is updated at less than every time step, and (iii) a posterior distribution over the possible latent variables generated from a hidden state of a fast updating recurrent neural network that is updated at every time step, the selecting comprising:
• determining whether criteria for updating the hidden state of the slow updating recurrent neural network that is updated at less than every time step are satisfied at the time step;
• when the criteria are satisfied, processing a slow updating input, comprising the hidden state of the fast updating recurrent neural network that is updated at every time step, to update the hidden state of the slow updating recurrent neural network;
• processing a fast updating input for the time step comprising (i) the observation, (ii) the hidden state of the slow updating recurrent neural network using the fast updating recurrent neural network, and (iii) data defining the first prior distribution over possible latent variables to update the hidden state of the fast updating recurrent neural network to generate an updated hidden state of the fast updating recurrent neural network, wherein the first prior distribution is generated from the hidden state of the slow updating recurrent neural network as of the preceding time step;
• after updating the hidden state of the fast updating recurrent neural network:
• generating, from the updated hidden state of the fast updating recurrent neural network, the posterior distribution over possible latent variables;
• sampling a latent variable from the posterior distribution;
• and
• selecting the action to be performed by the agent based on the latent variable sampled from the posterior distribution;
• and
• updating parameters of the fast updating recurrent neural network, comprising:
• generating, from the hidden state of the slow updating recurrent neural network, a second prior distribution over possible latent variables;
• and
• updating the parameters of the fast updating recurrent neural network based at least in part on a divergence between (i) the second prior distribution generated from the hidden state of the slow updating recurrent neural network and (ii) the posterior distribution generated from the updated hidden state of the fast updating recurrent neural network.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent interacting with an environment, the operations comprising, at each of a plurality of time steps:

• receiving an observation characterizing a current state of the environment at the time step;
• and
• selecting an action to be performed by the agent in response to the observation based on (i) the observation, (ii) a first prior distribution over possible latent variables generated from a hidden state of a slow updating recurrent neural network that is updated at less than every time step, and (iii) a posterior distribution over the possible latent variables generated from a hidden state of a fast updating recurrent neural network that is updated at every time step, the selecting comprising:
• determining whether criteria for updating the hidden state of the slow updating recurrent neural network that is updated at less than every time step are satisfied at the time step;
• when the criteria are satisfied, processing a slow updating input, comprising the hidden state of the fast updating recurrent neural network that is updated at every time step, to update the hidden state of the slow updating recurrent neural network;
• processing a fast updating input for the time step comprising (i) the observation, (ii) the hidden state of the slow updating recurrent neural network using the fast updating recurrent neural network, and (iii) data defining the first prior distribution over possible latent variables to update the hidden state of the fast updating recurrent neural network to generate an updated hidden state of the fast updating recurrent neural network, wherein the first prior distribution is generated from the hidden state of the slow updating recurrent neural network as of the preceding time step;
• after updating the hidden state of the fast updating recurrent neural network:
• generating, from the updated hidden state of the fast updating recurrent neural network, the posterior distribution over possible latent variables;
• sampling a latent variable from the posterior distribution;
• and
• selecting the action to be performed by the agent based on the latent variable sampled from the posterior distribution;
• and
• updating parameters of the fast updating recurrent neural network, comprising:
• generating, from the hidden state of the slow updating recurrent neural network, a second prior distribution over possible latent variables;
• and
• updating the parameters of the fast updating recurrent neural network based at least in part on a divergence between (i) the second prior distribution generated from the hidden state of the slow updating recurrent neural network and (ii) the posterior distribution generated from the updated hidden state of the fast updating recurrent neural network.