Performing batched training for machine-learning pipelines
AssigneeInternational Business Machines Corporation
InventorsMartin Hirzel, Kiran A. Kate, Avraham Ever Shinnar
Life of this patent
Estimating the term…
Abstract
A computer-implemented method according to one embodiment includes identifying a machine learning pipeline and a plurality of training data batches; creating a plurality of tasks, based on the machine learning pipeline; and determining an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach.
Claims20
A computer-implemented method, comprising:
- identifying a machine learning pipeline and a plurality of training data batches;
- creating a plurality of tasks, based on the machine learning pipeline, the tasks including a transform task that includes an instance where an operator takes training data of one of the training data batches as input and outputs computed data, the tasks including a partial-fit task that includes an instance where an operator is trained partially using a single batch of the plurality of training data batches;
- and
- determining an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach, wherein the partial-fit task is prioritized over the transform task such that the partial-fit task is placed before the transform task within the determined order.
A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising:
- identifying, by the one or more processors, a machine learning pipeline and a plurality of training data batches;
- creating, by the one or more processors, a plurality of tasks, based on the machine learning pipeline, the tasks including a transform task that includes an instance where an operator takes training data of one of the training data batches as input and outputs computed data, the tasks including a partial-fit task that includes an instance where an operator is trained partially using a single batch of the plurality of training data batches;
- and
- determining, by the one or more processors, an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach, wherein the partial-fit task is prioritized over the transform task such that the partial-fit task is placed before the transform task within the determined order.
A system, comprising:
- a processor;
- and
- logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
- identify a machine learning pipeline and a plurality of predefined training data batches;
- create a plurality of tasks, based on the machine learning pipeline, the tasks including a transform task that includes an instance where an operator takes training data of one of the training data batches as input and outputs computed data, the tasks including a partial-fit task that includes an instance where an operator is trained partially using a single batch of the plurality of training data batches;
- and
- determine an order in which the plurality of tasks is executed, utilizing a resource usage-aware approach, wherein the partial-fit task is prioritized over the transform task such that the partial-fit task is placed before the transform task within the determined order.
Dependencies are inferred from the claim text.