Method and system for task planning for visual room rearrangement under partial observability

A robotic agent for task planning for visual room rearrangement under partial observability, the system comprising:

• a memory storing instructions;
• one or more Input/Output (I/O) interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
• capture an egocentric view of a room in an untidy current state comprising one or more objects among a plurality of objects in the room misplaced from associated receptacles, wherein the robotic agent is currently at a random location in the room with partial observability of the room;
• identify an unseen object list by comparing i) a visible object list obtained from the egocentric view and ii) an object list of the plurality of objects created by the robotic agent in accordance with a user-specified tidy goal state, wherein an unseen object from the unseen object list is an object outside the robotic agent's present field of view, or an object placed within a closed receptacle;
• trigger a large language model (LLM) based search network to predict a most probable receptacle for each unseen object from among the unseen object list, wherein the LLM based search network applies a two steps search technique comprising:
• filtering out, via a sorting network (SRTN), a set of receptacles from an object-receptacle list generated during learning phase of the robotic agent, wherein the filtering is in context of the unseen object list, wherein the filtered out set of receptacles belong to an implausible class, wherein the implausible class is a class in which the object is least probable to be identified;
• and
• identifying the most probable receptacle for each unseen object based on a probability score metric computed by a Scoring Network (SCN);
• wherein the robotic agent assumes the position of an unseen object to be the position of its most probable receptacle from the object-receptacle list created by the robotic agent in accordance with a user-specified tidy goal state, and wherein the robotic agent replans the search based on one or more conditions experienced during task planning execution;
• create a graph embedding from the spatial graph representation of the untidy current state and the user-specified tidy goal state via a graph representation network (GRN), trained using an encoder and a decoder, to generate meaningful embeddings from the untidy current state and the user-specified tidy goal state graph that incorporates a residual relative path length notion between every pair of nodes associated with the untidy current state and the user-specified tidy goal state;
• and
• perform task planning for visual room rearrangement via a Parameterized Deep-Q Network (P-DQN) with hybrid action space comprising a discrete action, denoting an index of a selected unseen object or the probable receptacle, followed by a continuous parameter which signifies a location for object placement or receptacle search, wherein the P-DQN uses a proxy reward network trained with episodic reward, generated from the stepwise hierarchical dense reward and overall episodic path length, and wherein the graph embedding enables state space of the P-DQN to understand semantic and geometric information of the untidy current state and the user-specified tidy goal state.

One or more non-transitory computer readable mediums storing instructions, which when executed by a hardware processor, cause the hardware processor to perform actions comprising:

• capturing, by a robotic agent, an egocentric view of a room in an untidy current state comprising one or more objects among a plurality of objects in the room misplaced from associated receptacles, wherein the robotic agent is currently at a random location in the room with partial observability of the room;
• identifying, by the robotic agent, an unseen object list by comparing i) a visible object list obtained from the egocentric view and ii) an object list of the plurality of objects created by the robotic agent in accordance with a user-specified tidy goal state, wherein an unseen object from the unseen object list is an object outside the robotic agent's present field of view, or an object placed within a closed receptacle;
• triggering, by the robotic agent, a large language model (LLM) based search network to predict a most probable receptacle for each unseen object from among the unseen object list, wherein the LLM based search network applies a two steps search technique comprising:
• filtering out, via a sorting network (SRTN), a set of receptacles from an object-receptacle list generated during learning phase of the robotic agent, wherein the filtering is in context of the unseen object list, wherein the filtered out set of receptacles belong to an implausible class, wherein the implausible class is a class in which the object is least probable to be identified;
• and
• identifying the most probable receptacle for each unseen object based on a probability score metric computed by a Scoring Network (SCN);
• wherein the robotic agent assumes the position of an unseen object to be the position of its most probable receptacle from the object-receptacle list created by the robotic agent in accordance with a user-specified tidy goal state, and wherein the robotic agent replans the search based on one or more conditions experienced during task planning execution;
• creating, by the robotic agent, a graph embedding from the spatial graph representation of the untidy current state and the user-specified tidy goal state via a graph representation network (GRN), trained using an encoder and a decoder, to generate meaningful embeddings from the untidy current state and the user-specified tidy goal state graph that incorporates a residual relative path length notion between every pair of nodes associated with the untidy current state and the user-specified tidy goal state;
• and
• task planning, by the robotic agent, for visual room rearrangement via a Parameterized Deep-Q Network (P-DQN) with hybrid action space comprising a discrete action, denoting an index of a selected unseen object or the probable receptacle, followed by a continuous parameter which signifies a location for object placement or receptacle search, wherein the P-DQN uses a proxy reward network trained with episodic reward, generated from the stepwise hierarchical dense reward and overall episodic path length, and wherein the graph embedding enables state space of the P-DQN to understand semantic and geometric information of the untidy current state and the user-specified tidy goal state.