Training and/or utilizing machine learning model(s) for use in natural language based robotic control

A system comprising:

• a robot comprising one or more actuators and at least one vision component;
• and
• a controller coupled to the robot, the controller configured to:
• receive a free-form natural language instruction describing a task for the robot, the free-form natural language instruction generated based on user interface input provided by a user via one or more user interface input devices;
• process the free-form natural language instruction using a natural language instruction encoder, to generate a latent goal representation of the free-form natural language instruction;
• receive an instance of vision data, the instance of vision data generated by the at least one vision component of the robot, and the instance of vision data capturing at least part of an environment of the robot;
• generate output based on processing, using a goal-conditioned policy network, at least (a) the instance of vision data and (b) the latent goal representation of the free-form natural language instruction;
• control the one or more actuators of the robot based on the generated output, wherein control of the one or more actuators of the robot causes the robot to perform at least one action indicated by the generated output;
• receive a goal image describing an additional task for the robot;
• process the goal image using a goal image encoder, to generate a latent goal representation of the goal image;
• receive an additional instance of vision data, the additional instance of vision data generated by the at least one vision component of the robot, and the additional instance of vision data capturing at least part of the environment of the robot;
• generate additional output based on processing, using the goal-conditioned policy network, at least (a) the additional instance of vision data and (b) the latent goal representation of the goal image;
• and
• control the one or more actuators of the robot based on the generated additional output, wherein control of the one or more actuators of the robot causes the robot to perform at least one additional action indicated by the generated additional output.

A non-transitory computer readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• receive a free-form natural language instruction describing a task for a robot, the free-form natural language instruction generated based on user interface input provided by a user via one or more user interface input devices;
• process the free-form natural language instruction using a natural language instruction encoder, to generate a latent goal representation of the free-form natural language instruction;
• receive an instance of vision data, the instance of vision data generated by at least one vision component of the robot, and the instance of vision data capturing at least part of an environment of the robot;
• generate output based on processing, using a goal-conditioned policy network, at least (a) the instance of vision data and (b) the latent goal representation of the free-form natural language instruction;
• control one or more actuators of the robot based on the generated output, wherein control of the one or more actuators of the robot causes the robot to perform at least one action indicated by the generated output;
• receive a goal image describing an additional task for the robot;
• process the goal image using a goal image encoder, to generate a latent goal representation of the goal image;
• receive an additional instance of vision data, the additional instance of vision data generated by the at least one vision component of the robot, and the additional instance of vision data capturing at least part of the environment of the robot;
• generate additional output based on processing, using the goal-conditioned policy network, at least (a) the additional instance of vision data and (b) the latent goal representation of the goal image;
• and
• control the one or more actuators of the robot based on the generated additional output, wherein control of the one or more actuators of the robot causes the robot to perform at least one additional action indicated by the generated additional output.