System and method for robotic planning in partially observable environments using large language models

A controller for manipulating objects by a robot arm having a gripper, comprising:

• a sensor interface configured to generate and transmit states and current observations acquired by sensors;
• an interface circuit configured to receive the states and the current observations from the sensor interface and a task description provided by a user;
• a large language model (LLM) planner configured to acquire the states and the task description and generate an action sequence command that operates the robot arm with the gripper based on the task description, the current observations, historical information including historical actions and historical observations from previous steps, wherein the LLM planner reasons about uncertainty of completion of the task due to incomplete information based on the current observations, wherein the LLM planner transmits the generated action sequence command to the robot arm;
• a robot interface configured to generate and transmit commands to operate the robot arm in response to the action sequence from the LLM planner;
• and
• an LLM evaluator configured to acquire the states, the current observations, analyze the historical information including historical actions and historical observations from previous steps, evaluate missing information required to complete the task, evaluate new information acquired by the sensors, estimate a task-execution status after the robot arm operated the generated action sequence, and output updated information to the LLM planner.