US 12,743,627 B2 · Reinforcement learning with auxiliary tasks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The method includes: training an action selection policy neural network, and during the training of the action selection neural network, training one or more auxiliary control neural networks and a reward prediction neural network. Each of the auxiliary control neural networks is configured to receive a respective intermediate output generated by the action selection policy neural network and generate a policy output for a corresponding auxiliary control task. The reward prediction neural network is configured to receive one or more intermediate outputs generated by the action selection policy neural network and generate a corresponding predicted reward. Training each of the auxiliary control neural networks and the reward prediction neural network comprises adjusting values of the respective auxiliary control parameters, reward prediction parameters, and the action selection policy network parameters.

Claims20

Dependent claims

A method performed by one or more computers, the method comprising:

obtaining a plurality of online experience tuples characterizing interaction of an agent with an environment over a plurality of time steps while the agent is controlled using an action selection policy neural network;
training a state value neural network that is configured to process an input comprising an observation characterizing a state of the environment to generate an value estimate that defines an expected n-step return starting from the state characterized by the observation, wherein the training comprises:
training the state value neural network on the plurality of online experience tuples;
and
performing extra value function regression by training the state value neural network on a plurality of additional experience tuples stored in a replay memory, comprising:
randomly varying a temporal position of a truncation window over which n-step returns are computed from the plurality of additional experience tuples stored in the replay memory;
and
training the state value neural network using the n-step returns computed from the plurality of additional experience tuples stored in the replay memory;
and
training the action selection policy neural network using the state value neural network.

7 more elements

A system comprising:

one or more computers;
and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining a plurality of online experience tuples characterizing interaction of an agent with an environment over a plurality of time steps while the agent is controlled using an action selection policy neural network;
training a state value neural network that is configured to process an input comprising an observation characterizing a state of the environment to generate an value estimate that defines an expected n-step return starting from the state characterized by the observation, wherein the training comprises:
training the state value neural network on the plurality of online experience tuples;
and
performing extra value function regression by training the state value neural network on a plurality of additional experience tuples stored in a replay memory, comprising:
randomly varying a temporal position of a truncation window over which n-step returns are computed from the plurality of additional experience tuples stored in the replay memory;
and
training the state value neural network using the n-step returns computed from the plurality of additional experience tuples stored in the replay memory;
and
training the action selection policy neural network using the state value neural network.

10 more elements

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining a plurality of online experience tuples characterizing interaction of an agent with an environment over a plurality of time steps while the agent is controlled using an action selection policy neural network;
training a state value neural network that is configured to process an input comprising an observation characterizing a state of the environment to generate an value estimate that defines an expected n-step return starting from the state characterized by the observation, wherein the training comprises:
training the state value neural network on the plurality of online experience tuples;
and
performing extra value function regression by training the state value neural network on a plurality of additional experience tuples stored in a replay memory, comprising:
randomly varying a temporal position of a truncation window over which n-step returns are computed from the plurality of additional experience tuples stored in the replay memory;
and
training the state value neural network using the n-step returns computed from the plurality of additional experience tuples stored in the replay memory;
and
training the action selection policy neural network using the state value neural network.

7 more elements

Dependencies are inferred from the claim text.

Reinforcement learning with auxiliary tasks

Life of this patent

Reinforcement learning with auxiliary tasks

Life of this patent

Abstract

Claims20