Open-vocabulary robotic control using multi-modal language models

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving an observation image of an environment;
• receiving an input text sequence that describes a task to be performed by an agent in the environment and that references one or more objects in the environment;
• generating an object localization input that includes (i) a natural language phrase describing the one or more objects and (ii) the observation image;
• processing the object localization input using an object localization neural network to generate an object localization output that identifies respective locations of the one or more objects in the observation image;
• generating a policy input based on the observation image, the input text sequence, and the object localization output;
• processing the policy input using a policy neural network to generate a policy output that defines an action to be performed by the agent in response to the observation image;
• selecting an action to be performed by the agent using the policy output;
• and
• causing the agent to perform the selected action.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving an observation image of an environment;
• receiving an input text sequence that describes a task to be performed by an agent in the environment and that references one or more objects in the environment;
• generating an object localization input that includes (i) a natural language phrase describing the one or more objects and (ii) the observation image;
• processing the object localization input using an object localization neural network to generate an object localization output that identifies respective locations of the one or more objects in the observation image;
• generating a policy input based on the observation image, the input text sequence, and the object localization output;
• processing the policy input using a policy neural network to generate a policy output that defines an action to be performed by the agent in response to the observation image;
• selecting an action to be performed by the agent using the policy output;
• and
• causing the agent to perform the selected action.