Preemptive entropy reduction in vector embeddings to enhance semantic search
AssigneeIntuit Inc.
InventorsSumangal Mandal, Vaishali Gupta, Amit Kaushal
Life of this patent
Estimating the term…
Abstract
At least one processor may receive documents and a set of validation prompts, segment the documents into chunks, embed the chunks into a vector space, query a large language model (LLM) with the validation prompts, intercept chunk retrievals from the vector space in response to the validation prompts, map the retrieved chunks to their positions within the documents, calculate document coverage for the retrieved chunks, generate a report of the document coverage, and refine at least one of the validation prompts, document segmentation, or embedding process in response to the report to reduce entropy in the vector space.
Claims18
A method for preemptive entropy reduction in vector embeddings to enhance semantic search, comprising:
- receiving, by at least one processor, documents and a set of validation prompts;
- segmenting, by the at least one processor, the documents into chunks;
- embedding, by the at least one processor, the chunks into a vector space;
- querying, by the at least one processor, a large language model (LLM) with the validation prompts;
- intercepting, by the at least one processor, retrieved chunks of the chunks from the vector space in response to the validation prompts;
- mapping, by the at least one processor, the retrieved chunks to their positions within the documents;
- calculating, by the at least one processor, document coverage for the retrieved chunks, the document coverage indicating an amount of each document covered by the retrieved chunks;
- calculating, by the at least one processor, individual document entropies with respect to the chunks used in response to the validation prompts based on the document coverage;
- aggregating, by the at least one processor, the individual document entropies to calculate a total system entropy;
- generating, by the at least one processor, a report compiling coverage analysis and entropy calculations;
- and
- iteratively refining, by the at least one processor, at least one of the validation prompts, document segmentation, or embedding process in response to the document coverage and the entropy in the report to reduce the entropy in the vector space until the total system entropy reaches a predetermined acceptable threshold prior to deployment of the semantic search.
A system for preemptive entropy reduction in vector embeddings to enhance semantic search, comprising:
- an LLM server;
- a database server;
- a processing server configured to:
- receive documents and a set of validation prompts,
- segment the documents into chunks,
- embed the chunks into a vector space,
- transmit the validation prompts to the LLM server configured to process the validation prompts and generate queries for retrieving the chunks from the database server,
- intercept retrieved chunks of the chunks from the database server in response to the LLM server queries,
- map the retrieved chunks to their positions within the documents,
- calculate document coverage for the retrieved chunks, the document coverage indicating an amount of each document covered by the retrieved chunks,
- calculate individual document entropies with respect to the chunks used in response to the validation prompts based on the document coverage,
- aggregate the individual document entropies to calculate a total system entropy,
- generate a report compiling coverage analysis and entropy calculations, and
- iteratively refine at least one of the validation prompts, document segmentation, or embedding process in response to the document coverage and the entropy in the report to reduce the entropy in the vector space until the total system entropy reaches a predetermined acceptable threshold prior to deployment of the semantic search;
- and
- a user interface configured to display the report and receive refinement inputs.
Dependencies are inferred from the claim text.