Training sparse networks with discrete weight values

A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a machine-trained (MT) network, the program comprising sets of instructions for:

• propagating a plurality of inputs through the MT network to generate an output for each of the inputs, wherein (i) each of the inputs is associated with an expected output, (ii) the MT network uses a plurality of network parameters to process the inputs, and (iii) each network parameter of a set of the network parameters is defined during training as a probability distribution across a ternary set of possible values for the network parameter, the ternary set of possible values for each network parameter comprising the value zero, a positive value for the network parameter, and a negation of the positive value for the network parameter;
• calculating a value of a loss function for the MT network comprising (i) a first term that measures network error based on the expected outputs compared to the generated outputs and (ii) a second term that penalizes divergence of the probability distribution for each network parameter in the set of network parameters from a predefined probability distribution for the network parameter, wherein the predefined probability distribution for each of the network parameters comprises a probability value for each possible values for the network parameter, wherein the largest probability value for each network parameter is the probability value for the value zero;
• and
• using the calculated loss function value to train the network parameters, wherein the second term of the loss function and the predefined probability distributions having the largest probability value for the value zero results in a majority of the network parameters being trained to have a value of zero,
• wherein for each network parameter, the second term increases as the probability values for the positive value for the network parameter and the negation of the positive value for the network parameter in the probability distribution for the network parameter increase,
• wherein for each network parameter of at least a particular subset of the network parameters, the first term decreases as the probability values for the positive value for the network parameter and the negation of the positive value for the network parameter in the probability distribution of the network parameter increase.