Data processing in a machine learning computer

A computer system comprising memory and one or more processors configured to:

• process data in a neural network to compute a network tensor comprising a plurality of tensor elements represented in an initial numerical format;
• compute a histogram of tensor elements;
• select a target numerical format, the target numerical format having a lower precision than the initial numerical format;
• evaluate a metric based on the histogram of tensor elements and the target numerical format, the metric indicating a degree of accuracy of a representation of the network tensor in the target numerical format;
• and
• based on the evaluated metric, convert the plurality of tensor elements from the initial numerical format to the target numerical format.

A non-transitory computer-readable storage medium storing computer program instructions which when executed on one or more processors perform the steps of:

• processing data in a neural network to compute a network tensor comprising a plurality of tensor elements represented in an initial numerical format;
• computing a histogram of tensor elements;
• selecting a target numerical format, the target numerical format having a lower precision than the initial numerical format;
• evaluating a metric based on the histogram of tensor elements and the target numerical format, the metric indicating a degree of accuracy of a representation of the network tensor in the target numerical format;
• and
• based on the evaluated metric, converting the plurality of tensor elements from the initial numerical format to the target numerical format.