Controlling robots using multi-modal language models

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for controlling a robot to perform a task, the operations comprising:

• receiving one or more observations that characterize a state of an environment at a time step, wherein the environment that-comprises one or more objects;
• receiving an input text sequence that describes the task to be performed by the robot in the environment and that references the one or more objects in the environment;
• generating an encoded representation of the input text sequence in an embedding space;
• generating a corresponding encoded representation of each of the one or more observations in the embedding space;
• generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation;
• processing the sequence of input tokens using a language model neural network to autoregressively generate an output text sequence that specifies a high-level plan for the robot at the time step, wherein the high-level plan at the time step defines a plurality of operations to be performed by the robot and an order of performance of the plurality of operations by the robot in response to the one or more observations that characterize the state of the environment at the time step;
• processing at least the output text sequence using a policy neural network to map the high-level plan to one or more actions to be performed by robot hardware of the robot in response to the one or more observations that characterize the state of the environment at the time step;
• and
• controlling the robot to perform the task based on the one or more actions.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for controlling a robot to perform a task, the operations comprising:

• receiving one or more observations that characterize a state of an environment at a time step, wherein the environment that comprises one or more objects;
• receiving an input text sequence that describes the task to be performed by the robot in the environment and that references the one or more objects in the environment;
• generating an encoded representation of the input text sequence in an embedding space;
• generating a corresponding encoded representation of each of the one or more observations in the embedding space;
• generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation;
• processing the sequence of input tokens using a language model neural network to autoregressively generate an output text sequence that specifies a high-level plan for the robot at the time step, wherein the high-level plan at the time step defines a plurality of operations to be performed by the robot and an order of performance of the plurality of operations by the robot in response to the one or more observations that characterize the state of the environment at the time step;
• processing at least the output text sequence using a policy neural network to map the high-level plan to one or more actions to be performed by robot hardware of the robot in response to the one or more observations that characterize the state of the environment at the time step;
• and
• controlling the robot to perform the task based on the one or more actions.