Machine learning hardware having reduced precision parameter components for efficient parameter update

A method comprising:

• generating a first weight having a first format comprising a first number of bits based at least in part on a second weight having a second format comprising a second number of bits and a residual having a third format comprising a third number of bits, the second number of bits and the third number of bits each being less than the first number of bits;
• updating the second weight based at least in part on the first weight;
• and
• updating the residual based at least in part on the updated second weight and the first weight;
• and
• updating the first weight based at least in part on the updated second weight and the updated residual;
• wherein the method is performed at least in part by circuitry for a neural network.

An apparatus comprising at least one learner of a multiple learner system comprising a plurality of components, the at least one learner configured:

• to generate a portion of a gradient;
• to provide the portion of the gradient to at least one other component of the multiple learner system;
• and
• to obtain at least a portion of a weight from the at least one other component of the multiple learner system;
• and
• to update the portion of the gradient based at least in part on the obtained at least a portion of the weight.