Controlling agents using scene memory data

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of controlling an agent interacting with an environment, the operations comprising, at each of a plurality of time steps:

• receiving a current observation characterizing a current state of the environment as of the time step;
• generating an embedding of the current observation;
• processing scene memory data comprising embeddings of prior observations received at prior time steps using an encoder neural network, wherein the encoder neural network is configured to apply an encoder self-attention mechanism to the scene memory data to generate an encoded representation of the scene memory data;
• processing the encoded representation of the scene memory data and the embedding of the current observation using a decoder neural network to generate an action selection output, wherein the decoder neural network comprises a decoder attention block that is configured to, for each of one or more attention heads:
• generate a decoder query from the embedding of the current observation;
• generate decoder values from the encoded representation of the scene memory data;
• generate decoder keys from the encoded representation of the scene memory data;
• and
• apply a decoder attention mechanism that uses the decoder query as a query input, the decoder values as a value input, and the decoder keys as a key input to generate a decoder attention output;
• selecting, based on the action selection output and in accordance with a control policy for the agent, an action from a set of possible actions to be performed by the agent in response to the current observation;
• and
• causing the agent to perform the selected action.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations of controlling an agent interacting with an environment, the operations comprising, at each of a plurality of time steps:

• receiving a current observation characterizing a current state of the environment as of the time step;
• generating an embedding of the current observation;
• processing scene memory data comprising embeddings of prior observations received at prior time steps using an encoder neural network, wherein the encoder neural network is configured to apply an encoder self-attention mechanism to the scene memory data to generate an encoded representation of the scene memory data;
• processing the encoded representation of the scene memory data and the embedding of the current observation using a decoder neural network to generate an action selection output, wherein the decoder neural network comprises a decoder attention block that is configured to, for each of one or more attention heads:
• generate a decoder query from the embedding of the current observation;
• generate decoder values from the encoded representation of the scene memory data;
• generate decoder keys from the encoded representation of the scene memory data;
• and
• apply a decoder attention mechanism that uses the decoder query as a query input, the decoder values as a value input, and the decoder keys as a key input to generate a decoder attention output;
• selecting, based on the action selection output and in accordance with a control policy for the agent, an action from a set of possible actions to be performed by the agent in response to the current observation;
• and
• causing the agent to perform the selected action.