Version control interface supporting time travel access of a data lake
OwnerAbhishek GUPTA, Christos KARAMANOLIS +2inventors
InventorsAbhishek GUPTA, Christos KARAMANOLIS, Richard P. SPILLANE, Marin NOZHCHEV
Life of this patent
Estimating the term…
Abstract
A version control interface provides for time travel with metadata management under a common transaction domain as the data. Examples generate a time-series of master branch snapshots for data objects stored in a data lake, with the snapshot comprising a tree data structure such as a hash tree and associated with a time indication. Readers select a master branch snapshot from the time-series, based on selection criteria (e.g., time) and use references in the selected master branch snapshot to read data objects from the data lake. This provides readers with a view of the data as of a specified time.
Claims20
A method of providing a version control interface for accessing a data lake, the method comprising:
- generating a time-series of master branch snapshots for data objects stored in the data lake, each master branch snapshot providing an overlay data structure for accessing the data objects, each master branch snapshot associated with a unique identifier and a time indication identifying a creation time of the master branch snapshot, wherein the sets of the data objects differ for different ones of the master branch snapshots, wherein generating the time-series of master branch snapshots comprises providing concurrency control to coordinate transactions of metadata for the set of the data objects with transactions of the data objects;
- based on at least a first selection criteria, selecting a first master branch snapshot from the time-series of master branch snapshots;
- reading, by a first reader, the data objects from the data lake using references in the first master branch snapshot;
- based on at least a second selection criteria, selecting a second master branch snapshot from the time-series of master branch snapshots, wherein the second master branch snapshot is associated with a different time indication than the first master branch snapshot;
- and
- reading, by a second reader, concurrently with reading by the first reader, the data objects from the data lake using references in the second master branch snapshot.
A computer system providing a version control interface for accessing a data lake, the computer system comprising:
- a processor;
- and
- a non-transitory computer readable medium having stored thereon program code executable by the processor, the program code causing the processor to:
- generate a time-series of master branch snapshots for data objects stored in the data lake, each master branch snapshot providing an overlay data structure for accessing the data objects, each master branch snapshot associated with a unique identifier and a time indication identifying a creation time of the master branch snapshot, wherein the sets of the data objects differ for different ones of the master branch snapshots, wherein generating the time-series of master branch snapshots comprises providing concurrency control to coordinate transactions of metadata for the set of the data objects with transactions of the data objects;
- based on at least a first selection criteria, select a first master branch snapshot from the time-series of master branch snapshots;
- read, by a first reader, the data objects from the data lake using references in the first master branch snapshot;
- based on at least a second selection criteria, select a second master branch snapshot from the time-series of master branch snapshots, wherein the second master branch snapshot is associated with a different time indication than the first master branch snapshot;
- and
- read, by a second reader, concurrently with reading by the first reader, the data objects from the data lake using references in the second master branch snapshot.
A non-transitory computer storage medium having stored thereon program code executable by a processor, the program code embodying a method comprising:
- generating a time-series of master branch snapshots for data objects stored in a data lake, each master branch snapshot providing an overlay data structure for accessing the data objects, each master branch snapshot associated with a unique identifier and a time indication identifying a creation time of the master branch snapshot, wherein the sets of the data objects differ for different ones of the master branch snapshots;
- based on at least a first selection criteria, selecting a first master branch snapshot from the time-series of master branch snapshots, wherein generating the time-series of master branch snapshots comprises providing concurrency control to coordinate transactions of metadata for the set of the data objects with transactions of the data objects;
- reading, by a first reader, the data objects from the data lake using references in the first master branch snapshot;
- based on at least a second selection criteria, selecting a second master branch snapshot from the time-series of master branch snapshots, wherein the second master branch snapshot is associated with a different time indication than the first master branch snapshot;
- and
- reading, by a second reader, the data objects from the data lake using references in the second master branch snapshot.
Dependencies are inferred from the claim text.