Shipping of data through etl stages
AssigneeInternational Business Machines Corporation
InventorsKalyanji Chintakayala, Krishna K. Bonagiri, Eric A. Jacobson
Life of this patent
Estimating the term…
Abstract
Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.
Claims1
A computer-implemented method for shipping data between stages of an extract, transform, and load (ETL) process, the method comprising:
- receiving, by a stage of the ETL process, column data;
- ascertaining, by the stage, a size of the column data received;
- and
- in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
- saving, by the stage, the column data;
- and
- creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data, wherein the created data locator is a data structure that includes metadata, the metadata comprising (i) a hash key, (ii) a source indicator, and (iii) a timestamp, wherein the hash key corresponds to the column data associated with the data locator, the source indicator corresponds to a source of origin, and the timestamp indicates when the data was placed in the stage, and wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.
Dependencies are inferred from the claim text.