Method and apparatus for pipeline parallelism compiling

A computing device comprising:

• one or more processors;
• and
• a memory storing instructions that, when executed by one or more processors, cause the computing device to:
• receive a source program associated with training of a machine learning model;
• receive, based on the source program, a plurality of operation groups comprising operations executed on training data of the machine learning model, wherein the plurality of operation groups comprises:
• one or more forward propagation operation groups associated with a forward propagation process of the training data of the machine learning model, and
• one or more backward propagation operation groups associated with a backward propagation process of the training data of the machine learning model;
• generate a plurality of batches from the training data of the machine learning model, wherein each batch of the plurality of batches corresponds to a portion of the training data of the machine learning model;
• output, for each batch of the plurality of batches, a plurality of operation sets corresponding to the plurality of operation groups comprising:
• one or more forward propagation operation sets associated with the one or more forward propagation operation groups, and
• one or more backward propagation operation sets associated with the one or more backward propagation operation groups;
• determine an accelerator of a plurality of accelerators allocated to a corresponding operation set of the plurality of operation sets;
• determine a processing sequence of the corresponding operation set of the plurality of operation sets such that each operation set of the plurality of operation sets is processed in one accelerator of the plurality of accelerators;
• and
• execute a sequential process of each forward propagation operation set of a plurality of forward propagation operation sets associated with a specific batch of the plurality of batches in different accelerators of the plurality of accelerators according to a first order.

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing device, cause the computing device to:

• receive a source program associated with training of a machine learning model;
• receive, based on the source program, a plurality of operation groups comprising operations executed on training data of the machine learning model, wherein the plurality of operation groups comprises:
• one or more forward propagation operation groups associated with a forward propagation process of the training data of the machine learning model, and
• one or more backward propagation operation groups associated with a backward propagation process of the training data of the machine learning model;
• generate a plurality of batches from the training data of the machine learning model, wherein each batch of the plurality of batches corresponds to a portion of the training data of the machine learning model;
• output, for each batch of the plurality of batches, a plurality of operation sets corresponding to the plurality of operation groups comprising:
• one or more forward propagation operation sets associated with the one or more forward propagation operation groups, and
• one or more backward propagation operation sets associated with the one or more backward propagation operation groups;
• determine an accelerator of a plurality of accelerators allocated to a corresponding operation set of the plurality of operation sets;
• determine a processing sequence of the corresponding operation set of the plurality of operation sets such that each operation set of the plurality of operation sets is processed in one accelerator of the plurality of accelerators;
• and
• execute a sequential process of each forward propagation operation set of a plurality of forward propagation operation sets associated with a specific batch of the plurality of batches in different accelerators of the plurality of accelerators according to a first order.