Method and apparatus with prediction of training time of neural network model

An apparatus for predicting a training time, comprising:

• a memory storing a neural network;
• and
• at least one processor configured to:
• generate a first execution graph representing a distributed learning process of the neural network model, wherein the first execution graph includes a computation node representing a repeated computation of the neural network model based on a parallelism technique;
• generate a second execution graph based on the first execution graph and a target operator corresponding to the computation node, wherein the second execution graph includes an execution time of a hardware kernel performing the target operator;
• and predict a training time of the neural network model based on the second execution graph.

A method comprising:

• obtaining information representing a structure of a neural network model;
• dividing an operation of the neural network model into a plurality of repeated computations based on a parallelism technique and a hardware device to obtain a plurality of computation nodes;
• generating a first execution graph including the plurality of computation nodes based on the structure of the neural network model and the hardware device;
• generating a second execution graph based on the first execution graph and a hardware kernel of the hardware device, wherein the second execution graph includes a node representing the hardware kernel;
• profiling a target operator of the plurality of computation nodes based on the hardware kernel to obtain an execution time;
• and
• computing a predicted training time based on the second execution graph and the execution time.