Time estimator for deep learning architecture

A computer system for optimizing a neural network by estimating an inference time for each operator in the neural network, comprising:

• one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising:
• determining a benchmark time for at least one single-path architecture out of a plurality of single-path architectures associated with the neural network by sampling the at least one single-path architecture from the neural network, wherein the at least one single-path architecture comprises one or more operators;
• and
• based on the benchmark time for the at least one single-path architecture, determining an estimated inference time for an operator, wherein determining the estimated inference time for the operator comprises:
• applying an operator function, wherein the operator function comprises a function based on a difference between the benchmark time associated with the at least one single-path architecture and the estimated latency of the neural network.

A computer program product for optimizing a neural network by estimating an inference time for each operator in the neural network, comprising:

• one or more tangible computer-readable storage devices and program instructions stored on at least one of the one or more tangible computer-readable storage devices, the program instructions executable by a processor, the program instructions comprising:
• program instructions to determine a benchmark time for at least one single-path architecture out of a plurality of single-path architectures associated with the neural network by sampling the at least one single-path architecture from the neural network, wherein the at least one single-path architecture comprises one or more operators;
• and
• program instructions to determine, based on the benchmark time for the at least one single-path architecture, an estimated inference time for an operator, wherein determining the estimated inference time for the operator comprises:
• program instructions to apply an operator function, wherein the operator function comprises a function based on a difference between the benchmark time associated with the at least one single-path architecture and the estimated latency of the neural network.