Method and apparatus with distributed training of neural network

An apparatus comprising:

• a controller configured to,
• while training a neural network (NN) using a current training mode selected from a plurality of training modes for training of the NN, measure time data of a plurality of sub-operations for the training of the NN,
• based on the time data, determine a computation time to perform computation operations among the plurality of sub-operations and a communication time to perform communication operations among the plurality of sub-operations,
• based on a comparison result of the computation time and the communication time, select a next training mode from the plurality of training modes, and
• train the NN based on the next training mode;
• and
• processing modules configured to execute workloads corresponding to the plurality of sub-operations.

An apparatus comprising:

• a first network node and a second network node configured to train a neural network (NN),
• wherein the first network node comprises:
• a controller configured to,
• while training the NN using a current training mode selected from a plurality of training modes for training of the NN, measure time data of a plurality of sub-operations for the training of the NN,
• based on the time data, determine a computation time to perform computation operations among the plurality of sub-operations and a communication time to perform communication operations among the plurality of sub-operations,
• based on a comparison result of the computation time and the communication time, select a next training mode from the plurality of training modes, and
• train the NN based on the next training mode;
• and
• processing modules configured to execute workloads corresponding to the plurality of sub-operations.