US 2023/0418878 A1Published application

Multi-model enrichment memory and catalog for better search recall with granular provenance and lineage

AssigneeMicrosoft Technology Licensing, LLC

InventorsKevin Corley WONUS, Chelsea Ann VILLANUEVA, Samuel Robert LESTER

Life of this patent

Estimating the term…

Loading patent…

Abstract

A multi-model data store comprises an enrichment catalog and an enrichment memory graph. Raw data from a source raw datastore is enriched based on an enrichment function. Enriched data is stored in graph nodes with edge associations indicating the enrichment type and confidence level. The enriched data is also stored in the enrichment catalog with full breadcrumb provenance and lineage identifying, with cataloged enrichment steps, down to a record and field in the source raw datastore. Additional enriched values and associations between graph nodes can be entered in the graph by a user. Queries to the multi-model database begin in the graph and continue to the enrichment catalog. Query results include the full sequential chain of enrichments to source raw data records and fields, and may include dynamically generated virtual enrichment catalog records that link search values not found in the enrichment catalog to enrichment catalog records via the graph.

Claims20

Dependent claims

A system for maintaining provenance and lineage across a persistent multi-model datastore, the system comprising:

a processor;
and
a memory device, the memory device storing program code to be executed by the processor, the program code comprising:
a multi-model datastore manager configured to:
receive a first raw data value from a source raw datastore and a first enrichment type for the first raw data value;
query an enrichment memory graph of the persistent multi-model datastore for a data value matching the first raw data value and having an association to the first enrichment type in the enrichment memory graph;
an enrichment engine configured to:
in response to determining that a data value matching the first raw data value and having an association to the first enrichment type is not registered in the enrichment memory graph, generate a first enriched data value by executing an enrichment function of the first enrichment type on the first raw data value;
an enrichment memory graph constructor configured to:
register a node for the first raw data value and a node for the first enriched data value in the enrichment memory graph;
and
create an edge relationship between the first raw data value node and the first enriched data value node in the enrichment memory graph based on the first enrichment type;
and
an enrichment catalog constructor configured to:
build a first enrichment catalog record in an enrichment catalog stored in the persistent multi-model datastore including the first enriched data value, an identifier of the enrichment function of the first enrichment type, and a respective identifier for each of a source collection, a first source record, and a first source field storing the first raw data value in the source raw datastore.

12 more elements

A method for maintaining provenance and lineage across a persistent multi-model datastore with uniform representation of enrichments, the method comprising:

receiving a first raw data value from a source raw datastore and a first enrichment type for the first raw data value;
querying an enrichment memory graph of the persistent multi-model datastore for a data value matching the first raw data value and having an association to the first enrichment type in the enrichment memory graph;
in response to determining that a data value matching the first raw data value and having an association to the first enrichment type is not registered in the enrichment memory graph, generating a first enriched data value by executing an enrichment function of the first enrichment type on the first raw data value;
registering a node for the first raw data value and a node for the first enriched data value in the enrichment memory graph;
creating an edge relationship between the first raw data value node and the first enriched data value node in the enrichment memory graph based on the first enrichment type;
and
building a first enrichment catalog record in an enrichment catalog stored in the persistent multi-model datastore including the first enriched data value, an identifier of the enrichment function of the first enrichment type, and a respective identifier for each of a source collection, a first source record, and a first source field storing the first raw data value in the source raw datastore.

4 more elements

A computer-readable storage medium having program code recorded thereon that when executed by at least one processor causes the at least one processor to perform a method, the method comprising:

receiving a first raw data value from a source raw datastore and a first enrichment type for the first raw data value;
querying an enrichment memory graph of the persistent multi-model datastore for a data value matching the first raw data value and having an association to the first enrichment type in the enrichment memory graph;
in response to determining that a data value matching the first raw data value and having an association to the first enrichment type is not registered in the enrichment memory graph, generating a first enriched data value by executing an enrichment function of the first enrichment type on the first raw data value;
registering a node for the first raw data value and a node for the first enriched data value in the enrichment memory graph;
creating an edge relationship between the first raw data value node and the first enriched data value node in the enrichment memory graph based on the first enrichment type;
and
building a first enrichment catalog record in an enrichment catalog stored in the persistent multi-model datastore including the first enriched data value, an identifier of the enrichment function of the first enrichment type, and a respective identifier for each of a source collection, a first source record, and a first source field storing the first raw data value in the source raw datastore.

4 more elements

Dependencies are inferred from the claim text.