Controlling an agent to explore an environment using observation likelihoods
OwnerBilal Piot, Bernardo Avila Pires +1inventors
InventorsBILAL PIOT, BERNARDO AVILA PIRES, MOHAMMAD GHESHLAGHI AZAR
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling an agent. One of the methods includes, while training a neural network used to control the agent, generating a reward value for the training as a measure of the divergence between the likelihood of the further observation under first and second statistical models of the environment, the first statistical model and second model being based on respective first and second histories of past observations and actions, the most recent observation in the first history being more recent than the most recent observation in the second history.
Claims24
A method for generating actions to be performed by an agent interacting with an environment, the environment taking at successive times a corresponding one of a plurality of states,
- the method comprising successively modifying a neural network which is operative to generate the actions and defined by a plurality of network parameters, by, at each of a plurality of successive time steps:
- (a) using the neural network to generate an action based on a current observation of the environment at a current time;
- (b) causing the agent to perform the generated action on the environment;
- (c) obtaining a further observation of the state of the environment following the performance of the generated action by the agent;
- (d) generating a reward value as a measure of the divergence between (i) the likelihood of the further observation under a first statistical model of the environment and (ii) the likelihood of the further observation under a second statistical model of the environment, wherein:
- the first statistical model is based on a first history of past observations and actions,
- the second statistical model is based on a second history of past observations and actions, and
- the most recent observation in the first history is more recent than the most recent observation in the second history;
- and
- (e) modifying one or more said network parameters of the neural network based on the reward value.
(canceled)
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for generating actions to be performed by an agent interacting with an environment, the environment taking at successive times a corresponding one of a plurality of states,
- the operations comprising successively modifying a neural network which is operative to generate the actions and defined by a plurality of network parameters, by, at each of a plurality of successive time steps:
- (a) using the neural network to generate an action based on a current observation of the environment at a current time;
- (b) causing the agent to perform the generated action on the environment;
- (c) obtaining a further observation of the state of the environment following the performance of the generated action by the agent;
- (d) generating a reward value as a measure of the divergence between (i) the likelihood of the further observation under a first statistical model of the environment and (ii) the likelihood of the further observation under a second statistical model of the environment, wherein:
- the first statistical model is based on a first history of past observations and actions,
- the second statistical model is based on a second history of past observations and actions, and
- the most recent observation in the first history is more recent than the most recent observation in the second history;
- and
- (e) modifying one or more said network parameters of the neural network based on the reward value.
A training system implemented by one or more computers and for training a neural network which is operative to generate actions to be performed by an agent on an environment, the neural network being defined by a plurality of network parameters, and the environment taking successive ones of a plurality of states at successive times, the training system comprising:
- a predictor operative to generate first and second statistical models of the system based respectively on actions generated by the neural network and on first and second histories of past observations and actions, the most recent observation in the first history being more recent than the most recent observation in the second history;
- a reward generator operative to generate, from a further observation of the system after the agent has performed an action, a reward value as a measure of the difference between the likelihoods of the further observation under the first and second statistical models;
- and a neural network updater which updates the neural network based on the reward value.
Dependencies are inferred from the claim text.