Training network with discrete weight values

A method for configuring a machine-trained (MT) network comprising input nodes, output nodes, and interior nodes between the input and output nodes, wherein each node produces an output value and each interior node and output node receives as input values a set of output values of other nodes and applies weights to each received input value, wherein the weights are configurable parameters for training, the method comprising:

• propagating a set of inputs through the MT network to generate a set of outputs, each input having a corresponding expected output;
• calculating a value of a continuously-differentiable augmented loss function that combines (i) a measurement of a difference between each output and its corresponding expected output and (ii) a term that biases training of the weights towards a set of discrete values, the set of discrete values for each weight comprising zero, a positive value for the weight, and a negation of the positive value for the weight, wherein the term has a rate of change of zero for a particular weight when the particular weight is equidistant between two adjacent discrete values for the particular weight;
• and
• training the weights by (i) back propagating a gradient of the continuously-differentiable augmented loss function at the calculated value, (ii) back propagating the calculated value through the MT network to determine, for each weight, a rate of change in the calculated value relative to a rate of change in the weight, and (iii) modifying each particular weight according to the determined rate of change for the particular weight.

A method for configuring a machine-trained (MT) network comprising input nodes, output nodes, and interior nodes between the input and output nodes, wherein the nodes are arranged in layers comprising an input node layer, one or more intermediate node layers, and an output node layer, wherein each layer has an associated value, wherein each node produces an output value and each interior node and output node receives as input values a set of output values of other nodes and applies weights to each received input value, wherein the weights are configurable parameters for training, the method comprising:

• propagating a set of inputs through the MT network to generate a set of outputs, each input having a corresponding expected output;
• calculating a value of a continuously-differentiable augmented loss function that combines (i) a measurement of a difference between each output and its corresponding expected output and (ii) a term that biases training of the weights towards a set of discrete values, wherein the set of discrete values comprises a separate set for each layer, wherein the set of discrete values for each layer comprises the associated value, 0, and the negative of the associated value;
• and
• training the weights by back propagating a gradient of the continuously-differentiable augmented loss function at the calculated value.

A non-transitory machine readable medium storing a program which when executed by at least one processing unit configures a machine-trained (MT) network comprising input nodes, output nodes, and interior nodes between the input and output nodes, wherein each node produces an output value and each interior node and output node receives as input values a set of output values of other nodes and applies weights to each received input value, wherein the weights are configurable parameters for training, the program comprising sets of instructions for:

• propagating a set of inputs through the MT network to generate a set of outputs, each input having a corresponding expected output;
• calculating a value of a continuously-differentiable augmented loss function that combines (i) a measurement of a difference between each output and its corresponding expected output and (ii) a term that biases training of the weights towards a set of discrete values, the set of discrete values for each weight comprising zero, a positive value for the weight, and a negation of the positive value for the weight, the term comprising a function of each of the weights that is (i) continuously differentiable and (ii) equal to zero for a particular weight when the particular weight equals any one of the discrete values for the weight;
• and
• training the weights by back propagating a gradient of the continuously-differentiable augmented loss function at the calculated value.

A non-transitory machine readable medium storing a program which when executed by at least one processing unit configures a machine-trained (MT) network comprising input nodes, output nodes, and interior nodes between the input and output nodes, wherein each node produces an output value and each interior node and output node receives as input values a set of output values of other nodes and applies weights to each received input value, wherein the weights are configurable parameters for training, the program comprising sets of instructions for:

• propagating a set of inputs through the MT network to generate a set of outputs, each input having a corresponding expected output;
• calculating a value of a continuously-differentiable augmented loss function that combines (i) a measurement of a difference between each output and its corresponding expected output and (ii) a term that biases training of the weights towards a set of discrete values, the set of discrete values for each weight comprising zero, a positive value for the weight, and a negation of the positive value for the weight, wherein the term has a rate of change of zero for a particular weight when the particular weight is equidistant between two adjacent discrete values for the particular weight;
• and
• training the weights by (i) back propagating a gradient of the continuously-differentiable augmented loss function at the calculated value, (ii) back propagating the calculated value through the MT network to determine, for each weight, a rate of change in the calculated value relative to a rate of change in the weight, and (iii) modifying each particular weight according to the determined rate of change for the particular weight.

A non-transitory machine readable medium storing a program which when executed by at least one processing unit configures a machine-trained (MT) network comprising input nodes, output nodes, and interior nodes between the input and output nodes, wherein the nodes are arranged in layers comprising an input node layer, one or more intermediate node layers, and an output node layer, wherein each layer has an associated value, wherein each node produces an output value and each interior node and output node receives as input values a set of output values of other nodes and applies weights to each received input value, wherein the weights are configurable parameters for training, the program comprising sets of instructions for:

• propagating a set of inputs through the MT network to generate a set of outputs, each input having a corresponding expected output;
• calculating a value of a continuously-differentiable augmented loss function that combines (i) a measurement of a difference between each output and its corresponding expected output and (ii) a term that biases training of the weights towards a set of discrete values, wherein the set of discrete values comprises a separate set for each layer, wherein the set of discrete values for each layer comprises the associated value, 0, and the negative of the associated value;
• and
• training the weights by back propagating a gradient of the continuously-differentiable augmented loss function at the calculated value.