Exploiting input data sparsity in neural network compute units

A method for performing computations using a hardware circuit configured to implement a neural network comprising a plurality of neural network layers, the method comprising:

• receiving, by a controller of the circuit, a batch of inputs for processing through a first neural network layer;
• generating, by the controller, a compressed representation of inputs based on a respective value of each input in the batch of inputs;
• storing, using an address register of the circuit, a plurality of addresses, wherein each address of the plurality of addresses is an address of a location in memory that stores an input in the compressed representation of inputs;
• using control signals generated by the controller, retrieving each input in the compressed representation of inputs from a corresponding location in memory based on a respective address of the corresponding location;
• and
• after retrieving each respective input, processing the respective input through the first neural network layer at least by multiplying the input with a corresponding weight value for the first neural network layer to generate an output for the first neural network layer.

Non-transitory machine-readable storage devices for storing instructions that are executable by one or more processing devices to cause performance of operations for performing neural network computations using a circuit configured to implement a neural network comprising a plurality of neural network layers, the operations comprising:

• receiving, by a controller of the circuit, a batch of inputs for processing through a first neural network layer;
• generating, by the controller, a compressed representation of inputs based on a respective value of each input in the batch of inputs;
• storing, using an address register of the circuit, a plurality of addresses, wherein each address of the plurality of addresses is an address of a location in memory that stores an input in the compressed representation of inputs;
• using control signals generated by the controller, retrieving each input in the compressed representation of inputs from a corresponding location in memory based on a respective address of the corresponding location;
• and
• after retrieving each respective input, processing the respective input through the first neural network layer, at least by multiplying the input with a corresponding weight value for the first neural network layer to generate an output for the first neural network layer.