US 2025/0085982 A1 · PIPELINE CONFIGURATION SUPPORT

Abstract

Data processing tools often use pipeline-based workflows, which consist of a sequence of operations. Each operation is configured according to configuration settings provided by a pipeline developer. The operations may use other software components such as frameworks that may also be configured. An application developer that defines a series of operations to be performed to achieve a desired result is able to provide the configuration settings for the operations. However, the application developer may not have the expertise to efficiently define configuration settings for the underlying frameworks. As discussed herein, a pipeline configuration system is used to generate configuration settings for frameworks used by a pipeline based on configuration settings for the operations of the pipeline. The operation configuration may include non-transformable properties, transformable properties, and internal properties. The pipeline configuration system may primarily modify the transformable properties.

Claims20

Dependent claims

A system comprising:

a memory that stores instructions;
and
one or more processors coupled to the memory and configured to execute the instructions to perform operations comprising:
accessing operation configuration data for an operation of a data processing pipeline, the operation configuration data comprising non-transformable properties and a target cost;
based on the target cost, determining internal properties comprising a number of processors to use for the operation;
generating framework configuration data for a framework underlying the operation, the framework configuration data comprising the non-transformable properties and the internal properties;
and
configuring the framework using the generated framework configuration data.

5 more elements

Dependencies are inferred from the claim text.

Pipeline configuration support

Life of this patent

Pipeline configuration support

Life of this patent

Abstract

Claims20