US 9,542,461 B2Granted

Enhancing performance of extract, transform, and load (ETL) jobs

AssigneeInternational Business Machines Corporation

InventorsManish A. Bhide, Krishna K. Bonagiri, Srinivas K. Mittapalli, Sumit Negi

Life of this patent

Estimating the term…

Loading patent…

Abstract

A computer receives one or more files having configuration information that includes data that defines a plurality of stages of an extract, transform, and load (ETL) job, wherein the plurality of stages comprise a read stage that is preceded by a write stage, and wherein the read stage reads data from a source location, and wherein the data that is read or a modified version of the data that is read is being written by the write stage that writes data to the source location. The computer replaces the read stage with a decompressor stage. The computer replaces the write stage with a compressor stage. The computer executes the decompressor stage and compressor stage on a field-programmable gate array that is programmatically customized with data compression and data decompression functionality to enhance the performance of the ETL job.

Claims16

Dependent claims

A computer program product for accelerating a read stage and a write stage of an extract, transform, load job comprising:

a computer readable hardware storage medium and program instructions stored on the computer readable hardware storage medium, the program instructions comprising:
program instructions to receive a file having configuration information that includes data that defines a plurality of stages of at least one extract, transform, and load (ETL) job;
program instructions to create a configuration file, wherein the configuration file is associated with the file, and wherein the configuration file includes one or more compression parameters, and wherein each compression parameter of the one or more compression parameters is selected from the group consisting of a compressor stage or a decompressor stage;
program instructions to identify a read stage that is preceded by a write stage within the file, wherein the read stage reads data from a source location, and wherein the data that is read or a modified version of the data that is read is being written by the write stage that writes data to the source location, and wherein the read stage and write stage each have a first parameter that represents a name of a stage, and wherein the first parameter is associated with a second parameter that represents a type of stage;
program instructions to add the identified read stage to the configuration file;
program instructions, in response to identifying the read stage that is preceded by the write stage within the file, to replace the read stage, with a decompressor stage that reads data from a hard disk, decompress the data from the hard disk and passes the data that is decompressed to another stage or to a computer program for further processing, wherein the decompressor stage is configured to be executed on a field-programmable gate array (FPGA), and wherein parameters for the read stage are a decompressor stage in the configuration file;
program instructions, in response to identifying the read stage that is preceded by the write stage within the file, to replace the write stage, with a compressor stage that compresses the data, and writes the data to a hard disk, wherein the compressor stage is configured to be executed on the FPGA, and wherein the parameters for the write stage are a compressor stage in the configuration file;
program instructions to receive one or more first indications from a user, wherein each first indication of the one or more first indications includes adding, modifying, or deleting a first stage of the plurality of stages;
program instructions to receive one or more second indications from the user, wherein each second indication of the one or more second indications include one or more parameters of a second stage of the plurality of stages in the configuration file;
and
program instructions to execute the decompressor stage and the compressor stage on the FPGA based on the configuration file.

8 more elements

A computer system for acceleration a read stage and a write stage of an extract, transform, and load job comprising:

one or more processors, one or more computer readable memories, one or more computer readable storage media and program instructions stored on the one or more storage media for execution by the one or more processors via the one or more memories, the program instructions comprising:
program instructions to receive a file having configuration information that includes data that defines a plurality of stages of at least one extract, transform, and load (ETL) job;
program instructions to create a configuration file, wherein the configuration file is associated with the file, and wherein the configuration file includes one or more compression parameters, and wherein each compression parameter of the one or more compression parameters is selected from the group consisting of a compressor stage or a decompressor stage;
program instructions to identify a read stage that is preceded by a write stage within the file, wherein the read stage reads data from a source location, and wherein the data that is read or a modified version of the data that is read is being written by the write stage that writes data to the source location, and wherein the read stage and write stage each have a first parameter that represents a name of a stage, and wherein the first parameter is associated with a second parameter that represents a type of stage;
program instructions to add the identified read stage to the configuration file;
program instructions, in response to identifying the read stage that is preceded by the write stage within the file, to replace the read stage, with a decompressor stage that reads data from a hard disk, decompress the data from the hard disk and passes the data that is decompressed to another stage or to a computer program for further processing, wherein the decompressor stage is configured to be executed on a field-programmable gate array (FPGA), and wherein parameters for the read stage are a decompressor stage in the configuration file;
program instructions, in response to identifying the read stage that is preceded by the write stage within the file, to replace the write stage, with a compressor stage that compresses the data, and writes the data to a hard disk, wherein the compressor stage is configured to be executed on the FPGA, and wherein the parameters for the write stage are a compressor stage in the configuration file;
program instructions to receive one or more first indications from a user, wherein each first indication of the one or more first indications includes adding, modifying, or deleting a first stage of the plurality of stages;
program instructions to receive one or more second indications from the user, wherein each second indication of the one or more second indications include one or more parameters of a second stage of the plurality of stages in the configuration file;
and
program instructions to execute the decompressor stage and the compressor stage on the FPGA based on the configuration file.

8 more elements

Dependencies are inferred from the claim text.