US 2026/0216870 A1Published application

Method for automatic knowledge refinement and task execution

AssigneeTata Consultancy Services Limited

InventorsSnehasis BANERJEE, Shivam SINGH, Karthik SWAMINATHAN, Nabanita DASH, Ramandeep SINGH, Mohan SRIDHARAN, Madhava KRISHNA

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

This disclosure relates generally to a method for automatic knowledge refinement and task execution in robotic agents using a domain-agnostic knowledge base. Conventional Large Language Model (LLM) based methods break down complex tasks into sub-tasks but may generate incorrect steps or refer to unavailable objects or actions. While the robotic agents leverages prior-domain specific-knowledge, the unavailability of such structured knowledge in many practical domains and difficulty in updating data reliably makes agent-based task execution problematic. The integration of the domain-agnostic knowledge base with a Multi-modal Language Models (MLM) or LLM enables the robotic agent to perform tasks that span multiple domains without the constraints imposed by traditional, domain-specific knowledge bases. Combining the domain-agnostic knowledge base and the MLM allows for the generation of refined action sequences that are adaptable to the nuances of complex tasks, thereby enhancing the robotic agent's operational versatility and effectiveness.

Claims15

Dependent claims

A processor implemented method for automatic knowledge refinement and task execution in robotic agents, the method comprising:

receiving via a robotic agent a task to be executed wherein the task is decomposed into a plurality of sub-task;
generating via a Multi-modal Language Model (MLM) or an LLM (Large Language Model) an action sequence from each of the plurality of sub-task;
checking via the robotic agent the action sequence across a domain-agnostic knowledge graph, wherein the domain-agnostic knowledge graph is constructed dynamically by performing a sequence of steps comprising:
receiving a plurality of entities and a plurality of properties from the robotic agent based on working environment;
demonstrating the plurality of entities as a plurality of nodes and the plurality of properties as a plurality of edges;
generating an initial seed knowledge base (KB) based on the plurality of nodes and the plurality of edges;
allocating weights to the plurality of edges based on the relationship between the plurality of entities;
generating a state graph using the plurality of edges and the plurality of nodes;
generating an attribute graph from the initial seed knowledge base and the state graph;
querying the MLM and the initial seed KB using a plurality of queries generated from the plurality of sub-tasks;
combining the MLM output with the multimodal data from a plurality of sensors of the robotic agent;
extracting a plurality of objects from the combined output;
updating the attribute graph, KB and the state graph using the extracted data;
and
generating the domain agnostic KB based on the updated initial seed KB and the updated attribute graph;
generating via the robotic agent, a refined action sequence based on the MLM output and the domain agnostic knowledge graph;
and
executing via the robotic agent the refined action sequence.

15 more elements

A system, comprising:

a memory storing instructions;
one or more communication interfaces;
and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive a task to be executed wherein the task is decomposed into a plurality of sub-task;
generate via an MLM (Multi-modal Language Model) or an LLM (Large Language Model) an action sequence from the plurality of sub-task;
check the action sequence across a domain-agnostic knowledge graph, wherein the domain-agnostic knowledge graph is constructed dynamically by performing a sequence of steps comprises:
receiving a plurality of entities and a plurality of properties from the robotic agent based on working environment;
demonstrating the plurality of entities as a plurality of nodes and the plurality of properties as a plurality of edges;
generating an initial seed knowledge base (KB) based on the plurality of nodes and the plurality of edges;
allocating weights to the plurality of edges based on the relationship between the plurality of entities;
generating a state graph using the plurality of edges and the plurality of nodes;
generating an attribute graph from the initial seed knowledge base and the state graph;
querying the MLM and the initial seed KB using a plurality of queries generated from the plurality of sub-tasks;
combining the MLM output with the multimodal data from a plurality of sensors of the robotic agent;
extracting a plurality of objects from the combined output;
updating the attribute graph, KB and the state graph using the extracted data;
and
generating the domain agnostic KB based on the updated initial seed KB and the updated attribute graph;
generate a refined action sequence based on the MLM output and the domain agnostic knowledge graph;
execute the refined action sequence by the robotic agent.

18 more elements

One or non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

receiving a task to be executed wherein the task is decomposed into a plurality of sub-task;
generating via an MLM (Multi-modal Language Model) or an LLM (Large Language Model) an action sequence from the plurality of sub-task;
checking the action sequence across a domain-agnostic knowledge graph, wherein the domain-agnostic knowledge graph is constructed dynamically by performing a sequence of steps comprises:
receiving a plurality of entities and a plurality of properties from the robotic agent based on working environment;
demonstrating the plurality of entities as a plurality of nodes and the plurality of properties as a plurality of edges;
generating an initial seed knowledge base (KB) based on the plurality of nodes and the plurality of edges;
allocating weights to the plurality of edges based on the relationship between the plurality of entities;
generating a state graph using the plurality of edges and the plurality of nodes;
generating an attribute graph from the initial seed knowledge base and the state graph;
querying the MLM and the initial seed KB using a plurality of queries generated from the plurality of sub-tasks;
combining the MLM output with the multimodal data from a plurality of sensors of the robotic agent;
extracting a plurality of objects from the combined output;
updating the attribute graph, KB and the state graph using the extracted data;
and
generating the domain agnostic KB based on the updated initial seed KB and the updated attribute graph;
generating a refined action sequence based on the MLM output and the domain agnostic knowledge graph;
executing the refined action sequence by the robotic agent.

14 more elements

Dependencies are inferred from the claim text.