Action selection for reinforcement learning using a manager neural network that generates goal vectors defining agent objectives

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the operations comprising, at each of a plurality of time steps:

• generating a latent representation, in a latent space, of a current state of the environment at the time step;
• generating, based at least in part on the latent representation of the current state of the environment at the time step, an initial goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment;
• pooling the initial goal vector for the time step and initial goal vectors for one or more preceding time steps to generate a final goal vector for the time step, comprising:
• generating the final goal vector for the time step by element-wise combining the initial goal vector for the time step and the initial goal vectors for one or more preceding time steps;
• generating a respective action score for each action in the predetermined set of actions based at least in part on the final goal vector for the time step;
• and
• selecting an action from the predetermined set of actions to be performed by the agent at the time step using the action scores.

A method performed by one or more data processing apparatus for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the method comprising, at each of a plurality of time steps:

• generating a latent representation, in a latent space, of a current state of the environment at the time step;
• generating, based at least in part on the latent representation of the current state of the environment at the time step, an initial goal vector that defines, in the latent state space, an objective to be accomplished as a result of actions performed by the agent in the environment;
• pooling the initial goal vector for the time step and initial goal vectors for one or more preceding time steps to generate a final goal vector for the time step, comprising:
• generating the final goal vector for the time step by element-wise combining the initial goal vector for the time step and the initial goal vectors for one or more preceding time steps;
• generating a respective action score for each action in the predetermined set of actions based at least in part on the final goal vector for the time step;
• and
• selecting an action from the predetermined set of actions to be performed by the agent at the time step using the action scores.