US 2025/0094438 A1Published application

Data flow optimization system and method

AssigneesInventec (Pudong) Technology Corporation, Inventec Corporation

InventorsYu-Lun CHANG, Shu-Huei YANG, Wei-Chao CHEN

Life of this patent

Estimating the term…

Loading patent…

Abstract

An optimization method for data flow includes converting a plurality of extract-transform-load (ETL) scripts into a data flow diagram, wherein the data flow diagram includes a plurality of data lineages, with each data lineage including a plurality of data nodes; performing a first merging process, which repeatedly identifies at least two data lineages with the same nodes and merges them into a data family until any two of the data lineages do not have the same nodes; performing a second merging process, which calculates a column similarity of a pair of node, wherein the pair of nodes includes two adjacent data nodes within a data lineage; when the column similarity exceeds a threshold, merging the two data nodes; when the column similarity of the pair of nodes falls below the threshold, the data flow diagram is optimized, which is then converted into ETL scripts accordingly.

Claims10

Dependent claims

A data flow optimization method, performed by a processor, comprising:

converting a plurality of extract-transform-load scripts into a data flow diagram, wherein the data flow diagram includes a plurality of data lineages, and each of the plurality of data lineages includes a plurality of data nodes connected in a series;
performing a first merging process to generate a plurality of data families, wherein the first merging process comprises:
repeatedly finding at least two data lineages from the plurality of data lineages that have an identical node;
and merging the at least two data lineages into one of the plurality of data families until any two of the plurality of data lineages do not have the identical node;
after the first merging process is completed, performing a second merging process on each of the plurality of data lineages, wherein the second merging process comprises:
calculating a column similarity of a node pair, wherein the node pair comprises two adjacent data nodes with the series;
when the column similarity exceeds a first threshold, merging the two adjacent data nodes into one node;
and
when the column similarity of the node pair does not exceed the first threshold, converting the data flow diagram back into an optimized script.

6 more elements

A data flow optimization system comprising:

a storage device configured to store a plurality of extract-transform-load scripts and a plurality of instructions;
and
a computing device electrically connected to the storage device and configured to execute the plurality of instructions to perform a plurality of operations, wherein the plurality of instructions comprises:
converting a plurality of extract-transform-load scripts into a data flow diagram, wherein the data flow diagram includes a plurality of data lineages, each of the plurality of data lineages includes a plurality of data nodes connected in a series;
performing a first merging process to generate a plurality of data families, wherein the first merging process comprises:
repeatedly finding at least two data lineages from the plurality of data lineages that have an identical node;
and merging the at least two data lineages into one of the plurality of data families until any two of the plurality of data lineages do not have the identical node;
after the first merging process is completed, performing a second merging process on each of the plurality of data lineages, wherein the second merging process comprises:
calculating a column similarity of a node pair, wherein the node pair comprises two adjacent data nodes with the series;
when the column similarity exceeds a first threshold, merging the two adjacent data nodes into one node;
and
when the column similarity of the node pair does not exceed the first threshold, converting the data flow diagram back into an optimized script.

9 more elements

Dependencies are inferred from the claim text.