Control and training of robots using large language models

A cobot comprising:

• one or more motors for causing motion of the cobot;
• one or more processors;
• and
• one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the cobot to perform one or more operations, the operations including:
• receiving a query from a user, the query formed to trigger the cobot to perform a target objective;
• communicating the query to a large language model (LLM), the LLM constrained to respond to requests to cause the cobot to perform a given objective with a set of human-readable discrete tasks to accomplish the given objective;
• receiving a target set of human-readable discrete tasks from the LLM, the target set of human-readable discrete tasks comprising a first portion of human-readable discrete tasks and a second portion of human-readable discrete tasks and the target set of human-readable discrete tasks corresponding to target computer-readable code that is configured to cause the cobot to perform the target objective;
• providing the target set of human-readable discrete tasks to the user;
• delaying execution of the target computer-readable code until receiving authorization from the user;
• receiving authorization from the user that the first portion of human-readable discrete tasks passes an audit by the user;
• and
• executing a portion of the target computer-readable code that corresponds to the first portion of human-readable discrete tasks to perform a portion of the target objective.

A cobot comprising:

• one or more motors for causing motion of the cobot;
• one or more processors;
• and
• one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the cobot to perform one or more operations, the operations including:
• traversing a region to capture an image of an item with which the cobot is able to interact;
• communicating the image of the item to a large language model (LLM) to identify the item;
• receiving a query from a user, the query formed to trigger the cobot to perform a target objective relative to the item;
• receiving information identifying a set of known locations within an environment, the information comprising coordinates and distances of the known locations from known objects;
• communicating the query to the LLM, the LLM constrained to respond to requests to cause the cobot to perform a given objective with a set of human-readable discrete tasks to accomplish the given objective relative to the item;
• receiving a target set of human-readable discrete tasks from the LLM, the target set of human-readable discrete tasks being based on the set of known locations and corresponding to target computer-readable code that is configured to cause the cobot to perform the target objective relative to the item;
• providing the target set of human-readable discrete tasks to the user;
• and
• delaying execution of the target computer-readable code until receiving authorization from the user.