Methods and apparatuses for training a neural network

A system comprising a neural network where the neural network is trained by:

• receiving an input data set at a layer of the neural network;
• performing a forward pass and a backward pass on the input data set to determine regular output data, wherein the quantized forward pass and the quantized backward pass comprise providing a quantized version of each parameter in each layer of the neural network;
• calculating a first loss associated with the regular output data;
• performing a quantized forward pass and a quantized backward pass on the input data set to determine quantized output data;
• calculating a second loss associated with the quantized output data;
• comparing the first loss to the second loss, wherein said comparing comprises determining a magnitude of a difference between the first loss and the second loss;
• and
• based on the comparison, determining whether to reduce the input data set to provide a reduced data set, wherein determining whether to reduce the input data set comprises determining to reduce the input data set based on the magnitude of the difference between the first loss and the second loss being below a threshold value or zero.

A network node for implementing training of a neural network, the network node comprising processing circuitry configured to:

• receive an input data set at a layer of the neural network;
• perform a forward pass and a backward pass on the input data set to determine regular output data, wherein the quantized forward pass and the quantized backward pass comprise providing a quantized version of each parameter in each layer of the neural network;
• calculate a first loss associated with the regular output data;
• perform a quantized forward pass and a quantized backward pass on the input data set to determine quantized output data;
• calculate a second loss associated with the quantized output data;
• compare the first loss to the second loss, by determining a magnitude of a difference between the first loss and the second loss;
• and
• based on the comparison, determine whether to reduce the input data set to provide a reduced data set, wherein the processing circuitry is configured to determine to reduce the input data set based on the magnitude of the difference between the first loss and the second loss being below a threshold value or zero.