Robot systems, methods, control modules, and computer program products that leverage large language models

A method of operation of a robot system to perform a work objective, wherein the robot system includes a robot body, the method comprising:

• capturing, by at least one sensor of the robot system, sensor data representing information about an environment of the robot body;
• generating, by at least one processor of the robot system, a natural language (NL) description of at least one aspect of the environment based at least in part on the sensor data, the NL description of at least one aspect of the environment including at least one noun corresponding to at least one object in the environment;
• providing a NL query to a large language model (LLM) module, the NL query including the NL description of at least one aspect of the environment, an NL description of a first work objective, an NL description of an instruction set executable by the robot system, and an NL request for a first task plan;
• accessing a first parameterized task plan based on the NL description of the first work objective, wherein the first parameterized task plan includes at least one variable to be populated by a noun;
• populating the first parameterized task plan with at least one noun from the NL description of at least one aspect of the environment, wherein populating the first parameterized task plan with at least one noun from the NL description of the environment includes replacing each variable in the first parameterized task plan with a corresponding noun from the NL description of the environment;
• receiving the first populated task plan from the LLM module, the first populated task plan expressed in NL;
• and
• executing the first populated task plan by the robot system to perform the first work objective.