US 12,762,423 B2Granted

Reinforcement learning using hindsight to model unpredictable aspects of the future

AssigneeGDM Holding LLC

InventorsDaniel Jarrett, Corentin Tallec, Florent Altché, Thomas Mesnard, Remi Munos, Michal Valko

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network used to select actions to be performed by an agent interacting with an environment. Implementations of the method model unpredictable aspects of the future, using hindsight. They use this information to disentangle inherently unpredictable, aleatoric variation, from epistemic uncertainty that arises from lack of knowledge of the environment. They then use the epistemic uncertainty, which relates to in principle predictable aspects of the environment, as a source of intrinsic reward to drive curiosity, i.e. exploration of the environment by the agent.

Claims20

Dependent claims

A computer-implemented method of selecting actions to be performed by an agent in an environment using a policy neural network, comprising:

obtaining an observation representation representing an observation of the environment at a time step;
processing the observation of the environment at the time step using the policy neural network to generate a policy output used to select an action to be performed by the agent at the time step;
processing the observation representation at the time step, an observation representation at a next time step, and an action representation representing the action performed at the time step, using a generator neural network to generate a latent representation for the next time step;
processing the observation representation at the time step, the action representation at the time step, and the latent representation for the next time step, using an environment model neural network to generate a reconstructed representation for the observation at the next time step;
training the environment neural network using a reconstruction objective that depends on a difference between a version of the observation representation at the next time step and the reconstructed representation for the observation at the next time step;
training the generator neural network using an independence objective that depends on a degree of independence between the latent representation for the next time step and each of i) the observation representation at the time step and ii) the action representation representing the action performed at the time step, to encourage the degree of independence;
determining an intrinsic reward using the reconstruction objective;
and
training the policy neural network based on the intrinsic reward using a reinforcement learning technique.

6 more elements

A system comprising:

one or more computers;
and
one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an agent in an environment using a policy neural network, the operations comprising:
obtaining an observation representation representing an observation of the environment at a time step;
processing the observation of the environment at the time step using the policy neural network to generate a policy output used to select an action to be performed by the agent at the time step;
processing the observation representation at the time step, an observation representation at a next time step, and an action representation representing the action performed at the time step, using a generator neural network to generate a latent representation for the next time step;
processing the observation representation at the time step, the action representation at the time step, and the latent representation for the next time step, using an environment model neural network to generate a reconstructed representation for the observation at the next time step;
training the environment neural network using a reconstruction objective that depends on a difference between a version of the observation representation at the next time step and the reconstructed representation for the observation at the next time step;
training the generator neural network using an independence objective that depends on a degree of independence between the latent representation for the next time step and each of i) the observation representation at the time step and ii) the action representation representing the action performed at the time step, to encourage the degree of independence;
determining an intrinsic reward using the reconstruction objective;
and
training the policy neural network based on the intrinsic reward using a reinforcement learning technique.

9 more elements

Dependencies are inferred from the claim text.