Robotic reasoning through planning with language models

A method implemented by one or more processors, the method comprising:

• identifying an instruction for a robot to perform a task in an environment, the instruction being a free-form natural language instruction;
• determining, based on processing sensor data from one or more sensors of the robot, textual feedback that describes a current state of the environment of the robot;
• processing the instruction and the textual feedback using a large language model (LLM) to generate LLM output that is dependent on the instruction and that indicates one or more sub-tasks for performing the task;
• identifying a robotic skill that is performable by the robot, and a textual skill description of the robotic skill;
• determining, based on comparing the LLM output to the skill description, to implement the robotic skill;
• and
• in response to determining to implement the robotic skill:
• causing the robot to implement the robotic skill in the environment.