Semantic object disambiguation and robot navigation using a hierarchical scene graph with sensorial constraints

A method for performing a task on objects in an environment by a robot, wherein the method uses a processor coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor carry out steps of the method, comprising:

• receiving a semantic instruction that includes spatial or functional context of the task;
• receiving, via one or more sensors, a sensorial perception of the environment corresponding to a current position of the robot;
• accessing a memory storing a semantic hierarchical scene graph representing the environment, the scene graph comprising a connected graph of nodes organized into a multilevel hierarchy including nodes associated with objects in the environment, wherein edges between nodes are subject to sensorial connectivity constraints such that an edge connects a first node to a second node only if the sensorial perception of the robot enables navigation from a location associated with the first node to a location associated with the second node without passing through a location associated with a third node;
• identifying a subgraph of the semantic hierarchical scene graph associated with the task by using a large language model (LLM) trained with machine learning to exhibit reasoning abilities, the identification being based on the task and the current position of the robot within the scene graph;
• and
• generating a control command based on the identified subgraph and the task, and providing the control command to a motor to navigate the robot within the environment.