Method and system for data provenance management in multi-layer systems
AssigneeYahoo! Inc.
InventorsChris Olston, Anish Das Sarma
Life of this patent
Estimating the term…
Abstract
Method, system, and programs for heterogeneous data management. Information from multiple data sources is first obtained. Data/metadata from each of the data sources is modeled based on the source and/or granularity information of the data/metadata to generate data/metadata models. The data/metadata from multiple data sources are integrated, by applying one or more processes to the data/metadata from different data sources based on the data/metadata models, to generate integrated data/metadata. A provenance representation for the integrated data/metadata is created tracing sources, granularities, and/or processes applied and archived for enabling an query associated with the integrated data/metadata.
Claims20
A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for heterogeneous data management, comprising the steps of:
- obtaining, via the communication platform, information from a plurality of data sources;
- modeling, by a data/metadata modeling unit, the data/metadata from each of the data sources based on provenance and/or granularity information associated with the data/metadata to generate data/metadata models;
- integrating, by a data/metadata integration unit, the data/metadata from the plurality of data sources, by applying one or more processes to the data/metadata from different data sources in accordance with the data/metadata models, to generate integrated data/metadata;
- creating, by a provenance representation generator, a provenance representation of the integrated data/metadata that traces the plurality of data sources, granularities, and the one or more processes applied to generate the integrated data/metadata from the plurality of data sources;
- receiving, by a data/metadata manager, a query originated from a user, regarding information associated with the integrated data/metadata;
- and
- providing, by the data/metadata manager, a response directed to the query derived based on the provenance representation of the integrated data/metadata.
A system for heterogeneous data management, comprising:
- a heterogeneous data/metadata input unit implemented on a processor and configured for obtaining, via a communication platform, information from a plurality of data sources;
- a data/metadata modeling unit implemented on the processor and configured for modeling the data/metadata from each of the data sources based on a source and/or granularity information associated with the data/metadata to generate data/metadata models;
- a data/metadata integration unit implemented on the processor and configured for integrating the data/metadata from the plurality of data sources, by applying one or more processes to the data/metadata from different data sources in accordance with the data/metadata models, to generate integrated data/metadata;
- a provenance representation generator implemented on the processor and configured for creating a provenance representation of the integrated data/metadata that traces the plurality of data sources, granularities, and the one or more processes applied to generate the integrated data/metadata from the plurality of data sources, wherein the provenance representation is stored in a memory coupled to the processor;
- a data/metadata manager implemented on the processor and configured for receiving a query, originated from a user, regarding information associated with the integrated data/metadata and providing a response directed to the query based on the provenance representation of the integrated data/metadata.
A machine-readable tangible and non-transitory medium having information for heterogeneous data management recorded thereon, wherein the information, when read by the machine, causes the machine to perform the following
- obtaining, via the communication platform, information from a plurality of data sources;
- modeling, by a data/metadata modeling unit, the data/metadata from each of the data sources based on source and/or granularity information associated with the data/metadata to generate data/metadata models;
- integrating, by a data/metadata integration unit, the data/metadata from the plurality of data sources, by applying one or more processes to the data/metadata from different data sources in accordance with the data/metadata models, to generate integrated data/metadata;
- creating, by a provenance representation generator, a provenance representation of the integrated data/metadata that traces the plurality of data sources, granularities, and the one or more processes applied to generate the integrated data/metadata from the plurality of data sources;
- receiving, by a data/metadata manager, a query originated from a user, regarding information associated with the integrated data/metadata;
- and
- providing, by the data/metadata manager, a response directed to the query based on the provenance representation of the integrated data/metadata.
A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for generating a data provenance representation, comprising the steps of:
- obtaining, via the communication platform, information from a plurality of data sources;
- modeling, by a data/metadata modeling unit, the data/metadata from each of the data sources based on a source and/or granularity information associated with the data/metadata to generate data/metadata models;
- integrating, by a data/metadata integration unit, the data/metadata from the plurality of data sources, by applying one or more processes to the data/metadata from different data sources in accordance with the data/metadata models, to generate integrated data/metadata;
- creating, by a provenance representation generator, a provenance representation of the integrated data/metadata that traces the plurality of data sources, granularities, and the one or more processes applied to generate the integrated data/metadata from the plurality of data sources;
- and
- archiving the provenance representation of the integrated data/metadata for enabling a query associated with the integrated data/metadata on source and/or granularity.
A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for processing data provenance query, comprising the steps of:
- enabling, by a data/metadata manager, an access to an archive storing one or more provenance representations of integrated data/metadata, wherein
- the integrated data/metadata is generated from a plurality of data sources, by applying one or more processes to the data/metadata from different data sources in accordance with data/metadata models, and
- each of the one or more provenance representations traces the plurality of data sources, granularities, and the one or more processes applied to generate the integrated data/metadata from the plurality of data sources;
- receiving, via the communication platform, a query originated from a user, regarding information associated with the integrated data/metadata;
- analyzing the query received to determine information queried;
- accessing the archive to retrieve at least one provenance representation;
- extracting the queried information from the at least one provenance representation;
- and
- generating a response based on the queried information as a response to the query.
Dependencies are inferred from the claim text.