Data filtering and optimization for etl (extract, transform, load) processes
AssigneeInternational Business Machines Corporation
InventorsYing Chen, Bin He, Rui Wang
Life of this patent
Estimating the term…
Abstract
A method and system are disclosed for use with an ETL (Extract, Transform, Load) process, comprising optimizing a filter expression to select a subset of data and evaluating the filter expression on the data after the extracting, before the loading, but not during the transforming of the ETL process. The method and system optimizes the filtering using a pipelined evaluation for single predicate filtering and an adaptive optimization for multiple predicate filtering. The adaptive optimization includes an initial phase and a dynamic phase.
Claims20
A computer-implemented method for use with an ETL (Extract, Transform, Load) process, comprising:
- optimizing a filter expression to select a subset of data;
- and
- evaluating the filter expression on the data i) after the extracting, ii) before the loading, and iii) not during the transforming of the ETL process.
An information warehouse system, the information warehouse system including a processor for ETL (Extract, Transform, Load) processing of data, the information warehouse system executing steps for:
- filtering the data, wherein the data are either extracted data or transformed data, to select a subset of the data;
- and
- optimizing the filtering using a pipelined evaluation for single predicate filtering and an adaptive optimization for multiple predicate filtering.
A computer program product for use with an ETL (Extract, Transform, Load) process, the computer program product comprising a computer useable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:
- evaluate a filter expression on data after the extracting, before the loading, and not during the transforming of the ETL process in order to select a subset of the data;
- and
- optimize the evaluation using a pipelined evaluation for a single predicate filter expression and an adaptive optimization for a multi-predicate filter expression.
Dependencies are inferred from the claim text.