Runtime optimization of computations of an artificial neural network compiled for execution on a deep learning accelerator

A computing apparatus, comprising:

• memory;
• and at least one microprocessor configured to:
• predict, for a first artificial neural network and by utilizing a second artificial neural network embedded in a runtime library and constructed independent of a description of the first artificial neural network, at least one execution option of execution options based on a pattern of data being processed by the first artificial neural network, wherein the at least one execution option comprises a hardware option optimized for processing the pattern of data;
• compile the first artificial neural network based on the runtime library executable on a device and controllable via the second artificial neural network;
• and convert data representative of a description of the first artificial neural network into a compiler output having instructions executable on the device to generate an output of the first artificial neural network, the instructions configured to call routines in the runtime library to cause the second artificial neural network to control execution of the runtime library based at least in part on the data being processed by the first artificial neural network by selecting from the execution options for the routines during execution of the instructions, wherein at least one of the routines is implemented based on the at least one execution option predicted, wherein the routines in the runtime library are not specific to the first artificial neural network, wherein the second artificial neural network is configured to receive data representative of features of the data being processed by the first artificial neural network and to identify, as an output, the execution options to optimize execution of the runtime library;
• and wherein the runtime library and/or the instructions measure performance of executing the compiler output on the device by recording an elapsed time of executing the compiler output and recording an elapsed time of executing a predetermined portion of the instructions on the device, and store performance data comprising the recorded elapsed times, the performance data being associated with the at least one execution option and with features of an input to the first artificial neural network, and the at least one microprocessor is further configured to use the performance data in machine learning to train the second artificial neural network to identify an execution option for subsequently processed inputs.

A non-transitory computer storage medium storing instructions which when executed by a computing apparatus cause the computing apparatus to perform operations comprising:

• predicting, for a first artificial neural network and by utilizing a second artificial neural network embedded in a runtime library and constructed independent of a description of the first artificial neural network, at least one execution option of execution options based on a pattern of data being processed by the first artificial neural network, wherein the at least one execution option comprises a hardware option optimized for processing the pattern of data;
• compiling the first artificial neural network based on the runtime library executable on a device and controllable via the second artificial neural network;
• and generating, by the computing apparatus from data representative of the description of the first artificial neural network, a compiler output having instructions executable to generate an output of the first artificial neural network, wherein at least one routine of a plurality of routines called in the runtime library to cause the second artificial neural network to control execution of the runtime library is implemented based on the at least one execution option predicted, wherein the routines in the runtime library are not specific to the first artificial neural network by selecting from the execution options for the routines during execution of the instructions, wherein the second artificial neural network is configured to receive data representative of features of the data being processed by the first artificial neural network and to identify, as an output, the execution options to optimize execution of the runtime library;
• and wherein the runtime library and/or the instructions measure performance of executing the compiler output on the device by recording an elapsed time of executing the compiler output and recording an elapsed time of executing a predetermined portion of the instructions on the device, and store performance data comprising the recorded elapsed times, the performance data being associated with the at least one execution option and with features of an input to the first artificial neural network, and use the performance data in machine learning to train the second artificial neural network to identify an execution option for subsequently processed inputs.