US 11,232,121 B2Granted

Method, apparatus, and computer-readable medium for data transformation pipeline optimization

OwnerValentin Moskovich, Kanika Dhuriainventors

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system, method and computer-readable medium for data transformation pipeline optimization including storing a transformation pipeline comprising data transformation components having associated input buffers and associated output buffers and being configured to apply data transformations to data in the associated input buffers and output results of the data transformations to the associated output buffers, receiving a data set comprising data fields, identifying satellite data fields for at least one transformation component, each satellite data field comprising data that is not utilized during the data transformations of the transformation component, and processing, by the data transformation engine, the data set through each transformation component in the transformation pipeline, the processing including removing satellite data fields from the input buffers, linking the removed satellite data fields to the remaining data in the input buffers, and applying the data transformations to data in input buffers and writing results to output buffers.

Claims19

Dependent claims

A method executed by a data transformation engine on one or more computing devices for data transformation pipeline optimization, the method comprising:

storing, by the data transformation engine, a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;
receiving, by the data transformation engine, a data set comprising a plurality of data fields;
identifying, by the data transformation engine, one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components, each satellite data field comprising data that is not utilized during the one or more data transformations applied by the at least one transformation component;
and
processing, by the data transformation engine, the data set through each transformation component in the transformation pipeline, the processing comprising:
removing any satellite data fields from the one or more input buffers associated with that transformation component;
linking the removed satellite data fields to the remaining data in the one or more input buffers by inserting one or more record identifiers of one or more records in the data set having at least one satellite data field in the one or more satellite data fields into the one or more input buffers and passing the one or more record identifiers to the one or more output buffers associated with that transformation component;
and
applying the one or more data transformations of that transformation component to remaining data in the one or more input buffers and writing a result data set to the one or more output buffers associated with that transformation component.

6 more elements

A data transformation engine for data transformation pipeline optimization, the data transformation engine comprising:

one or more processors;
and
one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
store a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;
receive a data set comprising a plurality of data fields;
identify one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components, each satellite data field comprising data that is not utilized during the one or more data transformations applied by the at least one transformation component;
and
process the data set through each transformation component in the transformation pipeline, the processing comprising:
removing any satellite data fields from the one or more input buffers associated with that transformation component;
linking the removed satellite data fields to the remaining data in the one or more input buffers by inserting one or more record identifiers of one or more records in the data set having at least one satellite data field in the one or more satellite data fields into the one or more input buffers and passing the one or more record identifiers to the one or more output buffers associated with that transformation component;
and
applying the one or more data transformations of that transformation component to remaining data in the one or more input buffers and writing a result data set to the one or more output buffers associated with that transformation component.

9 more elements

At least one non-transitory computer-readable medium storing computer-readable instructions that, when executed by a data transformation engine on one or more computing devices, cause the data transformation engine to:

store a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;
receive a data set comprising a plurality of data fields;
identify one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components, each satellite data field comprising data that is not utilized during the one or more data transformations applied by the at least one transformation component;
and
process the data set through each transformation component in the transformation pipeline, the processing comprising:
removing any satellite data fields from the one or more input buffers associated with that transformation component;
linking the removed satellite data fields to the remaining data in the one or more input buffers by inserting one or more record identifiers of one or more records in the data set having at least one satellite data field in the one or more satellite data fields into the one or more input buffers and passing the one or more record identifiers to the one or more output buffers associated with that transformation component;
and
applying the one or more data transformations of that transformation component to remaining data in the one or more input buffers and writing a result data set to the one or more output buffers associated with that transformation component.

6 more elements

A method executed by a data transformation engine on one or more computing devices for data transformation pipeline optimization, the method comprising:

storing, by the data transformation engine, a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;
receiving, by the data transformation engine, a data set comprising a plurality of data fields, each transformation component taking one or more data fields in the plurality of data fields as one or more input groups and producing one or more output groups;
identifying, by the data transformation engine, one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components by identifying one or more input data fields corresponding to each input group and one or more output data fields corresponding to each output group and identifying any satellite data fields for each input group and output group pair based at least in part on the one or more input data fields corresponding to that input group and the one or more output data fields corresponding to that output group, each satellite data field comprising data that is not utilized during the one or more data transformations applied by the at least one transformation component;
and
processing, by the data transformation engine, the data set through each transformation component in the transformation pipeline, the processing comprising:
removing any satellite data fields from the one or more input buffers associated with that transformation component;
linking the removed satellite data fields to the remaining data in the one or more input buffers;
and
applying the one or more data transformations of that transformation component to remaining data in the one or more input buffers and writing a result data set to the one or more output buffers associated with that transformation component.

6 more elements

A data transformation engine for data transformation pipeline optimization, the data transformation engine comprising:

one or more processors;
and
one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
store a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;
receive a data set comprising a plurality of data fields, each transformation component taking one or more data fields in the plurality of data fields as one or more input groups and producing one or more output groups;
identify one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components by identifying one or more input data fields corresponding to each input group and one or more output data fields corresponding to each output group and identifying any satellite data fields for each input group and output group pair based at least in part on the one or more input data fields corresponding to that input group and the one or more output data fields corresponding to that output group, each satellite data field comprising data that is not utilized during the one or more data transformations applied by the at least one transformation component;
and
process the data set through each transformation component in the transformation pipeline, the processing comprising:
removing any satellite data fields from the one or more input buffers associated with that transformation component;
linking the removed satellite data fields to the remaining data in the one or more input buffers;
and
applying the one or more data transformations of that transformation component to remaining data in the one or more input buffers and writing a result data set to the one or more output buffers associated with that transformation component.

9 more elements

store a transformation pipeline comprising a plurality of data transformation components, each data transformation component having one or more associated input buffers and one or more associated output buffers and being configured to apply one or more data transformations to data in the one or more associated input buffers and output results of the one or more data transformations to the one or more associated output buffers;
receive a data set comprising a plurality of data fields, each transformation component taking one or more data fields in the plurality of data fields as one or more input groups and producing one or more output groups;
identify one or more satellite data fields in the plurality of data fields for at least one transformation component in the plurality of transformation components by identifying one or more input data fields corresponding to each input group and one or more output data fields corresponding to each output group and identifying any satellite data fields for each input group and output group pair based at least in part on the one or more input data fields corresponding to that input group and the one or more output data fields corresponding to that output group, each satellite data field comprising data that is not utilized during the one or more data transformations applied by the at least one transformation component;
and
process the data set through each transformation component in the transformation pipeline, the processing comprising:
removing any satellite data fields from the one or more input buffers associated with that transformation component;
linking the removed satellite data fields to the remaining data in the one or more input buffers;
and
applying the one or more data transformations of that transformation component to remaining data in the one or more input buffers and writing a result data set to the one or more output buffers associated with that transformation component.

6 more elements

Dependencies are inferred from the claim text.