# US 2017/0371940 A1 · Shipping of data through etl stages

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0371940 A1
- Document ID: US20170371940A1
- Status: Published application
- Kind: A1
- Application number: 15/664,951
- Filed: 2017-07-31
- Published: 2017-12-28
- Earliest claimed: 2016-06-23
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Kalyanji Chintakayala; Krishna K. Bonagiri; Eric A. Jacobson
- Web page: https://patentgrep.com/patent/US20170371940A1

## CPC

- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06F 16/221: {Column-oriented storage; Management thereof}
- H04L 63/06: {for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

## Claims

### Claim 1

A computer-implemented method for shipping data between stages of an extract, transform, and load (ETL) process, the method comprising:
receiving, by a stage of the ETL process, column data;
ascertaining, by the stage, a size of the column data received; and
in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and
creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data, wherein the created data locator is a data structure that includes metadata, the metadata comprising (i) a hash key, (ii) a source indicator, and (iii) a timestamp, wherein the hash key corresponds to the column data associated with the data locator, the source indicator corresponds to a source of origin, and the timestamp indicates when the data was placed in the stage, and wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.
