Instruction pruning for neural networks

A system comprising:

• an arrangement of one or more processors;
• a computer storage arrangement coupled to the one or more processors and configured with program code that when executed by the one or more processors causes the one or more processors to perform operations including:
• selecting for a layer of a plurality of layers of a neural network, NN, model, a current dt-ds pair from a plurality of different dt-ds pairs supported by a target device, wherein each dt-ds pair specifies a data type, dt, and an associated circuit structure, ds, that is configurable to compute d×s operations in parallel on a set of input activations and a matrix of weights of the data type, d is a number of rows in a sub-matrix of the matrix of weights, s is a number of columns in the sub-matrix, and d×s≥ 1;
• selecting as pruned weights of the layer, one or more subsets of the weights, based at least on each subset of the one or more subsets including d×s weights in the matrix of weights of the layer;
• evaluating performance of a pruned version the NN model based on the data type of the current dt-ds pair of the layer and a pruned weight matrix that is the matrix of weights of the layer pruned according to the pruned weights of the layer;
• and
• compiling, in response to the performance satisfying a threshold, the pruned version of the NN model into an execution graph and configuration data for implementing, on the target device, the circuit structure of the current dt-ds pair of the layer.