Controlling agents over long time scales using temporal value transport

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a neural network system augmented with an external memory and used to control an agent interacting with an environment to perform a specified task, the operations comprising:

• causing the agent to perform a task episode in which the agent attempts to perform the specified task, comprising, at each of a sequence of time steps during the task episode:
• obtaining an observation characterizing a current state of the environment;
• for each of one or more read heads, reading data from the external memory, comprising:
• assigning a respective weight to the data written to the external memory at each preceding time step;
• and
• generating a read vector that combines the data written to the external memory at each preceding time step in accordance with the respective weights;
• writing data to the external memory;
• selecting, from the observation and using the neural network system, an action to be performed by the agent in response to the observation, comprising selecting the action using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;
• generating, using the neural network system, a value prediction representing a value of the environment being in the current state to successfully performing the specified task, comprising generating the value prediction using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;
• causing the agent to perform the selected action;
• and
• receiving an actual reward value as a result of the agent performing the selected action;
• generating a respective modified reward value for each of one or more particular time steps in the sequence, comprising:
• for each of one or more first time steps in the sequence:
• identifying one or more time steps that are before the first time step in the sequence;
• generating a respective modification for each identified time step from at least (i) a weight assigned to the identified time step while reading from the external memory at the first time step and (ii) a value prediction generated using the data read from the external memory at the first time step;
• and
• for each identified time step, applying the modification for the identified time step to the actual reward for the identified time step;
• and
• training, through reinforcement learning, the neural network system using at least the modified rewards for the particular time steps.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a neural network system augmented with an external memory and used to control an agent interacting with an environment to perform a specified task, the operations comprising:

• causing the agent to perform a task episode in which the agent attempts to perform the specified task, comprising, at each of a sequence of time steps during the task episode:
• obtaining an observation characterizing a current state of the environment;
• for each of one or more read heads, reading data from the external memory, comprising:
• assigning a respective weight to the data written to the external memory at each preceding time step;
• and
• generating a read vector that combines the data written to the external memory at each preceding time step in accordance with the respective weights;
• writing data to the external memory;
• selecting, from the observation and using the neural network system, an action to be performed by the agent in response to the observation, comprising selecting the action using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;
• generating, using the neural network system, a value prediction representing a value of the environment being in the current state to successfully performing the specified task, comprising generating the value prediction using at least (i) data read from the external memory at a preceding time step or (ii) data read from the external memory at the time step;
• causing the agent to perform the selected action;
• and
• receiving an actual reward value as a result of the agent performing the selected action;
• generating a respective modified reward value for each of one or more particular time steps in the sequence, comprising:
• for each of one or more first time steps in the sequence:
• identifying one or more time steps that are before the first time step in the sequence;
• generating a respective modification for each identified time step from at least (i) a weight assigned to the identified time step while reading from the external memory at the first time step and (ii) a value prediction generated using the data read from the external memory at the first time step;
• and
• for each identified time step, applying the modification for the identified time step to the actual reward for the identified time step;
• and
• training, through reinforcement learning, the neural network system using at least the modified rewards for the particular time steps.