US 11,537,890 B2 · Compressing weights for distributed neural networks

Abstract

Embodiments of the present disclosure include systems and methods for compressing weights for distributed neural networks. In some embodiments, a first network comprising a first set of weights is trained using a set of training data. A second network comprising a second set of weights is trained using the set of training data. A number of weights in the first set of weights is greater than a number of weights in the second set of weights. The first set of weights are adjusted based on a first loss determined by the first network and a second loss determined by the second network. The second set of weights are adjusted based on the first loss determined by the first network and the second loss determined by the second network. Values of the second set of weights are sent to a computing system.

Claims20

Dependent claims

A system comprising:

a set of processing units;
and
a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:
train a first network comprising a first set of weights using a set of training data;
train a second network comprising a second set of weights using the set of training data, wherein a number of weights in the first set of weights is greater than a number of weights in the second set of weights;
adjust the first set of weights based on a first loss determined by the first network and a second loss determined by the second network;
adjust the second set of weights based on the first loss determined by the first network and the second loss determined by the second network;
and
send values of the adjusted second set of weights to a computing system.

6 more elements

A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:

training a first network comprising a first set of weights using a set of training data;
training a second network comprising a second set of weights using the set of training data, wherein a number of weights in the first set of weights is greater than a number of weights in the second set of weights;
adjusting the first set of weights based on a first loss determined by the first network and a second loss determined by the second network;
adjusting the second set of weights based on the first loss determined by the first network and the second loss determined by the second network;
and
sending values of the adjusted second set of weights to a computing system.

3 more elements

Dependencies are inferred from the claim text.

Compressing weights for distributed neural networks

Life of this patent

Compressing weights for distributed neural networks

Life of this patent

Abstract

Claims20