Systems and methods for intelligently folding normalization and weight parameters during a training of an artificial neural network

A system for generating a hardware-executable artificial neural network for execution on a matrix multiply accelerator, the system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable storage media storing instructions that, when executed by the one or more processors, cause the system to:
• access a network graph representing a pre-trained artificial neural network comprising a plurality of nodes including at least one computational node configured to perform a weighted sum operation;
• transform the network graph into a transformed network graph by replacing the computational node with a system-generated node comprising computational operations corresponding to the computational node and normalization operations associated with a normalization layer;
• initialize normalization parameters of the system-generated node based on statistics derived from training data;
• retrain the artificial neural network using the transformed network graph, the retraining comprising folding parameters of the normalization operations and parameters of the computational operations to generate an integrated node having a folded weight matrix and a folded bias matrix;
• update the folded weight matrix and the folded bias matrix during training iterations;
• scale the folded weight matrix and the folded bias matrix based on a weight-representation heuristic;
• scale output activations associated with the integrated node based on a composite scaling factor that increases a signal-to-noise ratio of the output activations;
• generate a hardware-execution graph corresponding to the artificial neural network;
• and
• compile the hardware-execution graph into instructions executable by the matrix multiply accelerator.