Real-world robot control using transformer neural networks

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations for controlling an agent interacting with an environment, wherein the operations comprise:

• receiving a natural language text sequence that characterizes a task to be performed by the agent in the environment;
• generating an encoded representation of the natural language text sequence;
• and
• at each of a plurality of time steps:
• obtaining an observation image characterizing a state of the environment at the time step;
• processing the observation image using an image encoder neural network that is conditioned on the encoded representation of the natural language text sequence to generate an encoded representation of the observation image;
• generating, from at least the encoded representation of the observation image, a sequence of input tokens;
• processing the sequence of input tokens using a Transformer neural network to generate a policy output that defines an action to be performed by the agent in response to the observation image;
• selecting an action to be performed by the agent using the policy output;
• and
• causing the agent to perform the selected action.

One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations for controlling an agent interacting with an environment, wherein the operations comprise:

• receiving a natural language text sequence that characterizes a task to be performed by the agent in the environment;
• generating an encoded representation of the natural language text sequence;
• and
• at each of a plurality of time steps:
• obtaining an observation image characterizing a state of the environment at the time step;
• processing the observation image using an image encoder neural network that is conditioned on the encoded representation of the natural language text sequence to generate an encoded representation of the observation image;
• generating, from at least the encoded representation of the observation image, a sequence of input tokens;
• processing the sequence of input tokens using a Transformer neural network to generate a policy output that defines an action to be performed by the agent in response to the observation image;
• selecting an action to be performed by the agent using the policy output;
• and
• causing the agent to perform the selected action.