US 12,585,689 B1Granted

Multi-layered method of prefiltering and ranking for text based vector retrieval

AssigneeDell Products L.P.

InventorsMatthew Eaton, Shyam Singaraju, Anil Koluguri, David Sydow

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods, system, and non-transitory processor-readable storage medium for a text-based vector retrieval system are provided herein. An example method includes generating, by a text-based vector retrieval system, a set of topics Tx for a document set DX, generating for each document DN a set of topics TD, comparing the set of topics TD against the set of topics TX to generate a filtered set of topics TDFinal, comparing the topics TDFinal against final topics of all other documents to generate a set DNMatching, encoding each document from the document set into a vector database with corresponding DNMatching and TDFinal as metadata, grouping vectors into N search clusters, retrieving N topics from a query string to generate TQuery topics, comparing the TQuery topics against topics in each cluster to identify matching clusters having R matching topics, and calculating similarity scores for documents in the matching clusters.

Claims18

Dependent claims

A method for text-based vector retrieval, comprising:

generating, by a text-based vector retrieval system, a set of topics Tx for a document set DX;
generating, by the text-based vector retrieval system, for each document DN in the set DX, a set of topics TD;
comparing, by the text-based vector retrieval system, for each document, the set of topics TD against the set of topics TX to generate a filtered set of topics TDFinal;
comparing, by the text-based vector retrieval system, for each document DN in DX, the topics TDFinal against final topics of all other documents to generate a set DNMatching containing documents matching document DN when M number of topics match between DN and other documents, wherein M is a tunable hyperparameter;
encoding, by the text-based vector retrieval system, each document from the document set into a vector database with corresponding DNMatching and TDFinal as metadata;
grouping, by the text-based vector retrieval system, vectors into N search clusters, wherein each cluster comprises documents from corresponding DNMatching metadata;
retrieving, by the text-based vector retrieval system, N topics from a query string to generate TQuery topics;
comparing, by the text-based vector retrieval system, the TQuery topics against topics in each cluster to identify matching clusters having R matching topics;
and
calculating, by the text-based vector retrieval system, similarity scores for documents in the matching clusters, wherein the method is performed by at least one processing device comprising a processor coupled to a memory;
wherein calculating, by the text-based vector retrieval system, similarity scores for documents in the matching clusters comprises:
determining a cluster weight TClusterWeight based on matching topics;
calculating a vector similarity score VMatchValue between the query and documents, and
computing a final similarity score DSimilarity as a product of TClusterWeight and VMatchValue, wherein DSimilarity represents the final similarity score between a search query and a specific document in a document cluster.

11 more elements

A system comprising:

at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to generate, by a text-based vector retrieval system, a set of topics TX for a document set DX;
to generate, by the text-based vector retrieval system, for each document DN in the set DX, a set of topics TD;
to compare, by the text-based vector retrieval system, for each document, the set of topics TD against the set of topics TX to generate a filtered set of topics TDFinal;
to compare, by the text-based vector retrieval system, for each document DN in DX, the topics TDFinal against final topics of all other documents to generate a set DNMatching containing documents matching document DN when M number of topics match between DN and other documents, wherein M is a tunable hyperparameter;
to encode, by the text-based vector retrieval system, each document from the document set into a vector database with corresponding DNMatching and TDFinal as metadata;
to group, by the text-based vector retrieval system, vectors into N search clusters, wherein each cluster comprises documents from corresponding DNMatching metadata;
to retrieve, by the text-based vector retrieval system, N topics from a query string to generate TQuery topics;
to compare, by the text-based vector retrieval system, the TQuery topics against topics in each cluster to identify matching clusters having R matching topics;
and
to calculate, by the text-based vector retrieval system, similarity scores for documents in the matching clusters;
wherein calculating, by the text-based vector retrieval system, similarity scores for documents in the matching clusters comprises:
determining a cluster weight TClusterWeight based on matching topics;
calculating a vector similarity score VMatchValue between the query and documents, and
computing a final similarity score DSimilarity as a product of TClusterWeight and VMatchValue, wherein DSimilarity represents the final similarity score between a search query and a specific document in a document cluster.

13 more elements

A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to generate, by a text-based vector retrieval system, a set of topics TX for a document set DX;
to generate, by the text-based vector retrieval system, for each document DN in the set DX, a set of topics TD;
to compare, by the text-based vector retrieval system, for each document, the set of topics TD against the set of topics TX to generate a filtered set of topics TDFinal;
to compare, by the text-based vector retrieval system, for each document DN in DX, the topics TDFinal against final topics of all other documents to generate a set DNMatching containing documents matching document DN when M number of topics match between DN and other documents, wherein M is a tunable hyperparameter;
to encode, by the text-based vector retrieval system, each document from the document set into a vector database with corresponding DNMatching and TDFinal as metadata;
to group, by the text-based vector retrieval system, vectors into N search clusters, wherein each cluster comprises documents from corresponding DNMatching metadata;
to retrieve, by the text-based vector retrieval system, N topics from a query string to generate TQuery topics;
to compare, by the text-based vector retrieval system, the TQuery topics against topics in each cluster to identify matching clusters having R matching topics;
and
to calculate, by the text-based vector retrieval system, similarity scores for documents in the matching clusters;
wherein calculating, by the text-based vector retrieval system, similarity scores for documents in the matching clusters comprises:
determining a cluster weight TClusterWeight based on matching topics;
calculating a vector similarity score VMatchValue between the query and documents, and
computing a final similarity score DSimilarity as a product of TClusterWeight and VMatchValue, wherein DSimilarity represents the final similarity score between a search query and a specific document in a document cluster.

11 more elements

Dependencies are inferred from the claim text.