Back to results
US 10,853,338 B2Granted
Universal data pipeline
AssigneePalantir Technologies Inc.
InventorsJacob Meacham, Michael Harris, Gustav Brodman, Lynn Cuthriell, Hannah Korus, Brian Toth, Jonathan Hsiao, Mark Elliot, Brian Schimpf, Michael Garland, Evelyn Nguyen
Life of this patent
Estimating the term…
Abstract
A history preserving data pipeline computer system and method. In one aspect, the history preserving data pipeline system provides immutable and versioned datasets. Because datasets are immutable and versioned, the system makes it possible to determine the data in a dataset at a point in time in the past, even if that data is no longer in the current version of the dataset.
Claims20
A method comprising:
- at one or more computing devices comprising one or more processors and one or more storage media storing one or more computer programs executed by the one or more processors to perform the method, performing operations comprising:
- maintaining a build catalog comprising a plurality of build catalog entries, each build catalog entry comprising:
- an identifier of a version of a derived dataset corresponding to a build catalog entry,
- one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and
- a derivation program build dependency that is executable to generate the version of the derived dataset corresponding to the build catalog entry;
- creating a new version of a particular derived dataset based on providing one or more particular child dataset versions as input to executing a particular version of a particular derivation program;
- and
- adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of each of the one or more particular child dataset versions, and at least one identifier of one or more particular child dataset versions that were provided as input to the particular derivation program.
5 more elements
A computer system comprising:
- one or more hardware processors;
- one or more computer programs;
- and
- one or more storage media storing the one or more computer programs for execution by the one or more hardware processors, the one or more computer programs comprising instructions for performing operations comprising:
- maintaining a build catalog comprising a plurality of build catalog entries, each build catalog entry comprising:
- an identifier of a version of a derived dataset corresponding to a build catalog entry,
- one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and
- a derivation program build dependency that is executable to generate the version of the derived dataset corresponding to the build catalog entry;
- creating a new version of a particular derived dataset based on providing one or more particular child dataset versions as input to executing a particular version of a particular derivation program;
- and
- adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of each of the one or more particular child dataset versions, and at least one identifier of one or more particular child dataset versions that were provided as input to the particular derivation program.
8 more elements
Dependencies are inferred from the claim text.