Automated runtime configuration for dataflows

A system, comprising:

• a processor;
• and
• memory that stores program code executable by the processor to perform a method, the method comprising:
• receiving a dataflow plan comprising representations of data operations in a dataflow pipeline;
• determining, based on the received dataflow plan, metadata of the received dataflow plan;
• determining, based on the received dataflow plan, a feature set comprising a plurality of features;
• selecting a machine learning (ML) model from a plurality of ML models based on metadata of the received dataflow plan and metadata of a training set the selected ML model was trained on;
• providing the feature set to the selected ML model;
• receiving, from the selected ML model, a weight associated with a data operation of the data operations;
• and
• causing execution of the received dataflow plan by resources allocated based on the weight.

A method in a server, comprising:

• receiving a dataflow plan comprising representations of data operations in a dataflow pipeline;
• determining, based on the received dataflow plan, metadata of the received dataflow plan that comprises information regarding the data operations;
• selecting a machine learning (ML) model from a plurality of ML models based on the metadata of the dataflow plan and metadata of a training set the selected ML model was trained on;
• providing the generated metadata to the selected ML model;
• receiving, from the selected ML model, a weight associated with a data operation of the data operations;
• and
• causing execution of the received dataflow plan by resources allocated based on the weight.