US 12,743,426 B2Granted

Generating a unifiled knowledge graph from multimodal data sources using collaborative multi-agent large language models

AssigneeTata Consultancy Services Limited

InventorsAnkur Krishna, Surya Ardham, Chetan Premkumar Malhotra

Life of this patent

Estimating the term…

Loading patent…

Abstract

Existing multi-agent system are employed collaboratively, where each agent specialized in extracting entities, relations, or events from documents which is a more generalized collaborative framework without specific customizations. The disclosed method presents a collaborative multi-agent knowledge graph Retrieval-Augmented Generation (RAG) framework using a plurality of collaborative multi-agent LLMs for extracting information from a plurality of multimodal documents and constructing the unified knowledge graph to facilitate accurate and efficient query-based retrieval. A query generator agent generates a plurality of queries that aim to uncover all possible information present in the plurality of multimodal documents. A domain model is generated by a domain model generator agent based on the plurality of queries. The domain model populator agent populates the domain model with data extracted from the plurality of multimodal documents. A unified knowledge graph is constructed from ta plurality of populated domain models and organizes the extracted information for accurate retrieval.

Claims15

Dependent claims

A processor implemented method, the method comprising:

receiving, via one or more hardware processors, a plurality of multimodal documents, and a plurality of domain historical questions pertaining to the plurality of multimodal documents;
processing, via the one or more hardware processors, the plurality of multimodal documents to identify text and a plurality of images, using a python library;
chunking, via the one or more hardware processors, the text and the plurality of images into a plurality of chunks comprising a plurality of text chunks, and a plurality of image chunks;
and
generating, via the one or more hardware processors, a unified knowledge graph from the plurality of chunks, using a plurality of collaborative multi-agent large language models (LLMs), wherein the plurality of collaborative multi-agent LLMs comprises a query generator agent, a domain model generator agent, a domain model populator agent, and a knowledge graph curator agent, wherein generating the unified knowledge graph comprises:
(i) generating a plurality of queries associated with each chunk of the plurality of chunks, through the query generator agent comprises:
(a) building a query agent, by defining a plurality of query generator behavioral parameters, and a plurality of query generator functional specification parameters through an input JavaScript Object Notation (JSON) model, wherein the plurality of query generator behavioural parameters comprises a query generator role, an agent knowledge enriched using the plurality of domain historical questions, a query generator interface, and wherein the plurality of query generator functional specification parameters comprises a plurality of query generator actions, a plurality of tasks comprising a plurality of query generator tasks and a plurality of query generator evaluation tasks, a feedback from the plurality of collaborative multi-agent LLMs, and a plurality of query generator tools;
(b) formulating a query prompt template using a plurality of instructions, the plurality of chunks, an action, and the query agent;
(c) generating a query prompt using the query prompt template;
and
(d) feeding the query prompt to the query generator agent, to generate the plurality of queries associated with each chunk of the plurality of chunks;
(ii) generating a domain model associated with each chunk of the plurality of chunks based on the generated plurality of queries, through the domain model generator agent;
(iii) populating the generated domain model using each chunk of the plurality of chunks, to generate a populated domain model of a plurality of populated domain models for each chunk of the plurality of chunks, through the domain model populator agent;
and
(iv) generating the unified knowledge graph associated with each chunk of the plurality of chunks, using the plurality populated domain models, through the knowledge graph curator agent.

12 more elements

A system comprising:

a memory storing instructions;
one or more communication interfaces;
and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive a plurality of multimodal documents, and a plurality of domain historical questions pertaining to the plurality of multimodal documents;
process the plurality of multimodal documents to identify text and a plurality of images, using a python library;
chunk the text and the plurality of images into a plurality of chunks comprising a plurality of text chunks, and a plurality of image chunks;
and
generate a unified knowledge graph from the plurality of chunks, using a plurality of collaborative multi-agent large language models (LLMs), wherein the plurality of collaborative multi-agent LLMs comprises a query generator agent, a domain model generator agent, a domain model populator agent, and a knowledge graph curator agent, wherein generating the unified knowledge graph comprises:
(i) generating a plurality of queries associated with each chunk of the plurality of chunks, through the query generator agent comprises:
(a) building a query agent, by defining a plurality of query generator behavioral parameters, and a plurality of query generator functional specification parameters through an input JavaScript Object Notation (JSON) model, wherein the plurality of query generator behavioral parameters comprises a query generator role, an agent knowledge enriched using the plurality of domain historical questions, a query generator interface, and wherein the plurality of query generator functional specification parameters comprises a plurality of query generator actions, a plurality of query tasks comprising a plurality of query generator tasks and a plurality of query generator evaluation tasks, a feedback from the plurality of collaborative multi-agent LLMs, and a plurality of query generator tools;
(b) formulating a query prompt template using a plurality of instructions, the plurality of chunks, an action, and the query agent;
(c) generating a query prompt using the query prompt template;
and
(d) feeding the query prompt to the query generator agent, to generate the plurality of queries associated with each chunk of the plurality of chunks;
(ii) generating a domain model associated with each chunk of the plurality of chunks based on the generated plurality of queries, through the domain model generator agent;
(iii) populating the generated domain model using each chunk of the plurality of chunks, to generate a populated domain model of a plurality of populated domain models for each chunk of the plurality of chunks, through the domain model populator agent;
and
(iv) generating the unified knowledge graph associated with each chunk of the plurality of chunks, using the plurality populated domain models, through the knowledge graph curator agent.

16 more elements

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

receiving a plurality of multimodal documents, and a plurality of domain historical questions pertaining to the plurality of multimodal documents;
processing the plurality of multimodal documents to identify text and a plurality of images, using a python library;
chunking the text and the plurality of images into a plurality of chunks comprising a plurality of text chunks, and a plurality of image chunks;
and
generating a unified knowledge graph from the plurality of chunks, using a plurality of collaborative multi-agent large language models (LLMs), wherein the plurality of collaborative multi-agent LLMs comprises a query generator agent, a domain model generator agent, a domain model populator agent, and a knowledge graph curator agent, wherein generating the unified knowledge graph comprises:
(i) generating a plurality of queries associated with each chunk of the plurality of chunks, through the query generator agent comprises:
(a) building a query agent, by defining a plurality of query generator behavioral parameters, and a plurality of query generator functional specification parameters through an input JavaScript Object Notation (JSON) model, wherein the plurality of query generator behavioural parameters comprises a query generator role, an agent knowledge enriched using the plurality of domain historical questions, a query generator interface, and wherein the plurality of query generator functional specification parameters comprises a plurality of query generator actions, a plurality of tasks comprising a plurality of query generator tasks and a plurality of query generator evaluation tasks, a feedback from the plurality of collaborative multi-agent LLMs, and a plurality of query generator tools;
(b) formulating a query prompt template using a plurality of instructions, the plurality of chunks, an action, and the query agent;
(c) generating a query prompt using the query prompt template;
and
(d) feeding the query prompt to the query generator agent, to generate the plurality of queries associated with each chunk of the plurality of chunks;
(ii) generating a domain model associated with each chunk of the plurality of chunks based on the generated plurality of queries, through the domain model generator agent;
(iii) populating the generated domain model using each chunk of the plurality of chunks, to generate a populated domain model of a plurality of populated domain models for each chunk of the plurality of chunks, through the domain model populator agent;
and
(iv) generating the unified knowledge graph associated with each chunk of the plurality of chunks, using the plurality populated domain models, through the knowledge graph curator agent.

12 more elements

Dependencies are inferred from the claim text.