Robot systems, methods, control modules, and computer program products that leverage large language models

A robot system comprising:

• a robot body;
• at least one sensor to capture sensor data representing information about an environment of the robot body;
• at least one processor;
• and
• at least one non-transitory processor-readable medium communicatively coupled to the at least one processor and storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to:
• provide a natural language (NL) query to a large language model (LLM) module, the NL query including information about an environment of the robot body, an NL description of a work objective, an NL description of an instruction set which includes a plurality of instructions each selectively executable by the robot system, an NL request for a task plan, and an NL indication of a task plan step threshold indicating a maximum number of steps in the task plan;
• receive the task plan from the LLM module, the task plan expressed in NL and including a number of steps which does not exceed the task plan step threshold;
• for each step of the task plan which does not correspond to a single instruction in the instruction set, until each step of the task plan does correspond to a single instruction in the instruction set:
• provide an NL breakdown query to the LLM module, the NL breakdown query including an NL request to break down the step of the task plan into a plurality of component steps;
• receive the plurality of component steps from the LLM module expressed in NL;
• and
• incorporate the plurality of component steps from the LLM module into the task plan as steps;
• and
• cause the robot system to execute the task plan.

A robot system comprising:

• a robot body;
• at least one sensor to capture sensor data representing information about an environment of the robot body;
• at least one processor;
• and
• at least one non-transitory processor-readable medium communicatively coupled to the at least one processor and storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the at least one processor to:
• provide a first NL query to a large language model (LLM) module, the first NL query including information about an environment of the robot body, an NL description of a work objective, an NL description of an instruction set which includes a plurality of instructions each selectively executable by the robot system, an NL request for a task plan corresponding to a sequence of instructions from the instruction set, and an NL indication of a task plan instruction threshold for a maximum number of instructions in the sequence;
• receive an indication from the LLM module that generation of a task plan having a number of instructions which does not exceed the task plan instruction threshold has failed;
• provide a second NL query to the LLM module, the second NL query including a request to provide the task plan as a set of conceptual steps which does not exceed a task plan step threshold for a maximum number of steps;
• receive the task plan from the LLM module as the set of conceptual steps expressed in NL;
• for each step of the set of conceptual steps, until the task plan comprises only instructions specified in the instruction set:
• provide a respective NL breakdown query to the LLM module, the respective NL breakdown query including an NL request to break down the conceptual step of the task plan into a plurality of instructions from the set of instructions;
• receive either an NL description of the plurality of instructions from the LLM module, or an indication from the LLM module that generation of the plurality of instructions which does not exceed the task plan instruction threshold has failed;
• when an NL description of the plurality of instructions is received, incorporate the plurality of instructions into the task plan;
• when an indication from the LLM module that generation of the plurality of instructions which does not exceed the task plan instruction threshold has failed, provide a further NL breakdown query to the LLM module, the further NL breakdown query including a request to provide the conceptual step as a further set of conceptual steps which does not exceed a step threshold for a maximum number of steps;
• and
• cause the robot system to execute the task plan.