Customizable pipeline for integrating data
AssigneePalantir Technologies Inc.
InventorsAbhinav Khanna, Henry Tung, Lucas Ray, Stephen Yazicioglu, Alexander Martino
Life of this patent
Estimating the term…
Abstract
Systems, methods, and non-transitory computer readable media are provided for customizing pipelines for integrating data. A file to be ingested into a data analysis platform may be determined. The file type of the file may be detected. The file may be transformed based on the file type. The transformation may include applying a set of operations to the file. The set of operations may correspond to a pipeline of operations associated with the file type.
Claims14
A system comprising:
- one or more processors;
- and a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
- determining a file is to be ingested into a data analysis platform in response to a user selecting the file, wherein the file is of a file type;
- with existing detectors, detecting the file type based on structure information, pattern information, or actual information within the file;
- in response to the file being undetectable by the detectors, marking the file type as generic and creating a new detector to detect the file type;
- in response to one of the existing detectors or the new detector detecting the file type:
- generating, by the one of the existing detectors or the new detector, metadata including information relating to the file type or information relating to one or more operations to be performed based on the file type;
- identifying, by the one of the existing detectors or the new detector, a particular transformer, wherein the particular transformer executes at least a portion of the one or more operations based on the metadata;
- and
- passing the metadata to the particular transformer;
- and
- removing an effect resulting from a previous version of a pipeline of operations or from the one or more operations by constructing a delete pipeline.
A method implemented by a computing system including one or more processors and storage media storing machine-readable instructions, wherein the method is performed using the one or more processors, the method comprising:
- determining a file is to be ingested into a data analysis platform in response to a user selecting the file, wherein the file is of a file type;
- with existing detectors, detecting the file type based on structure information, pattern information, or actual information within the file;
- in response to the file being undetectable by the detectors, marking the file type as generic and creating a new detector to detect the file type;
- in response to one of the existing detectors or the new detector detecting the file type:
- generating, by the one of the existing detectors or the new detector, metadata including information relating to the file type or information relating to one or more operations to be performed based on the file type;
- identifying, by the one of the existing detectors or the new detector, a particular transformer, wherein the particular transformer executes at least a portion of the one or more operations based on the metadata;
- and
- passing the metadata to the particular transformer;
- and
- removing an effect resulting from a previous version of a pipeline of operations or from the one or more operations by constructing a delete pipeline.
A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
- determining a file is to be ingested into a data analysis platform in response to a user selecting the file, wherein the file is of a file type;
- with existing detectors, detecting the file type based on structure information, pattern information, or actual information within the file;
- in response to the file being undetectable by the detectors, marking the file type as generic and creating a new detector to detect the file type;
- in response to one of the existing detectors or the new detector detecting the file type:
- generating, by the one of the existing detectors or the new detector, metadata including information relating to the file type or information relating to one or more operations to be performed based on the file type;
- identifying, by the one of the existing detectors or the new detector, a particular transformer to perform at least a portion of the one or more operations based on the metadata;
- and
- passing the metadata to the particular transformer;
- and
- removing an effect resulting from a previous version of a pipeline of operations or from the one or more operations by constructing a delete pipeline.
Dependencies are inferred from the claim text.