Sparse activation-aware weight loading and inference for machine learning models

A system comprising:

• a neural network accelerator apparatus comprising
• one or more processors;
• one or more computer readable media storing processor executable instructions which, when executed using the one or more processors, perform operations comprising
• determining, based on first inference-time data and first weight data representing a first weight tensor, second inference-time data,
• determining, based on the second inference-time data and an activation function, first activation data representing a first activation tensor,
• determining, based on the first activation data and based on a first sparsity map indicating positions of the activation tensor to be used for calculation, compressed activation data representing a subset of values from the first activation tensor,
• determining, based on second weight data representing a second weight tensor and based on the first sparsity map, compressed weight data representing a subset of values from the second weight tensor,
• determining, based on the compressed activation data and the compressed weight data, third inference-time data.

A method comprising:

• generating a first sparsity map for a first activation tensor associated with a first layer of a first machine learning model, the first sparsity map indicating positions of non-zero values of the first activation tensor;
• identifying a first set of channels of a weight tensor, wherein each channel of the first set of channels is identified as corresponding to a respective non-zero value from the first sparsity map;
• loading the first set of channels of the weight tensor into memory;
• and
• generating, by the first machine learning model, output data based on the first set of channels and the first activation tensor.