Optimizations for machine learning data processing pipeline

A computer-implemented method, comprising:

• generating, based at least on one or more inputs received from a client device, a data processing pipeline configured to generate a machine learning model having a set of parameters for performing a task associated with an input dataset, the data processing pipeline including an orchestrator node, a preparator node, and a plurality of executor nodes, the preparator node configured to generate, based at least on the input dataset, a training dataset, the plurality of executor nodes configured to execute a plurality of machine learning trials, each of the plurality of machine learning trials being executed by at least applying, to the training dataset, a different type of machine learning model and/or a different sets of trial parameters, and the orchestrator node configured to identify, based at least on a result of the plurality of machine learning trials, the machine learning model having the set of parameters for performing the task;
• identifying one or more pools of machine learning trials, each pool of machine learning trials being assigned for execution at a single one of the plurality of executor nodes;
• and
• executing the data processing pipeline to generate the machine learning model for performing the task, the executing of the data processing pipeline includes executing the one or more pools of machine learning trials.

A non-transitory computer readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:

• generating, based at least on one or more inputs received from a client device, a data processing pipeline configured to generate a machine learning model having a set of parameters for performing a task associated with an input dataset, the data processing pipeline including an orchestrator node, a preparator node, and a plurality of executor nodes, the preparator node configured to generate, based at least on the input dataset, a training dataset, the plurality of executor nodes configured to execute a plurality of machine learning trials, each of the plurality of machine learning trials being executed by at least applying, to the training dataset, a different type of machine learning model and/or a different sets of trial parameters, and the orchestrator node configured to identify, based at least on a result of the plurality of machine learning trials, the machine learning model having the set of parameters for performing the task;
• identifying one or more pools of machine learning trials, each pool of machine learning trials being assigned for execution at a single one of the plurality of executor nodes;
• and executing the data processing pipeline to generate the machine learning model for performing the task, the executing of the data processing pipeline includes executing the one or more pools of machine learning trials.