# US 11,907,818 B2 · Compression of machine-learned models via entropy penalized weight reparameterization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,818 B2 - Document ID: US11907818B2 - Status: Granted - Kind: B2 - Application number: 18/165,211 - Filed: 2023-02-06 - Published: 2023-06-15 - Granted: 2024-02-20 - Earliest claimed: 2019-05-15 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Deniz Oktay; Saurabh Singh; Johannes Balle; Abhinav Shrivistava - Web page: https://patentgrep.com/patent/US11907818B2 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 20/00: Machine learning - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Example aspects of the present disclosure are directed to systems and methods that learn a compressed representation of a machine-learned model (e.g., neural network) via representation of the model parameters within a reparameterization space during training of the model. In particular, the present disclosure describes an end-to-end model weight compression approach that employs a latent-variable data compression method. The model parameters (e.g., weights and biases) are represented in a “latent” or “reparameterization” space, amounting to a reparameterization. In some implementations, this space can be equipped with a learned probability model, which is used first to impose an entropy penalty on the parameter representation during training, and second to compress the representation using arithmetic coding after training. The proposed approach can thus maximize accuracy and model compressibility jointly, in an end-to-end fashion, with the rate-error trade-off specified by a hyperparameter. ## Claims ### Claim 1 A computer-implemented method to learn a compressed representation of a machine-learned model via a representation within a reparameterization space, the method comprising: employing, by the one or more computing devices, the machine-learned model to generate a prediction based on a training input, wherein an uncompressed representation of the machine-learned model is characterized by a plurality of model parameters that are representable, within the reparameterization space, by a plurality of reparametrized representations; modifying, by the one or more computing devices, at least one data element of a set of data elements based on an evaluation of a loss function that is a function of the prediction, wherein the set of data elements includes at least the plurality of parameterized representations and one or more decoder models that are configured to transform the plurality of reparametrized representations into the plurality of model parameters; after modifying the at least one data element, generating, by the one or more computing devices, the compressed reparametrized representation of the machine-learned model based on applying one or more entropy coding compression techniques to the plurality of reparametrized representations. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 further comprising: obtaining, by one or more computing devices, data descriptive of the plurality of model parameters; defining, by the one or more computing devices, the plurality of reparametrized representations for the plurality of model parameters of the machine-learned model; and generating, by the one or more computing devices, the one or more decoder models. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein modifying the at least one data element comprises: modifying, by the one or more computing devices, at least one of the plurality of reparametrized representations based on the evaluation of the loss function. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein modifying the at least one data element comprises: modifying, by the one or more computing devices, at least one of the one or more decoder models based on the evaluation of the loss function. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein modifying the at least one data element comprises backpropagating, by the one or more computing devices, the evaluation of the loss function through the one or more decoder models with modification of the one or more decoder models; and after backpropagating, by the one or more computing devices, the evaluation of the loss function through the one or more decoder models, continuing to backpropagate, by the one or more computing devices, the evaluation of the loss function through the plurality of reparametrized representations with modification of the plurality of reparametrized representations. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: transmitting, by the one or more computing devices, data descriptive of the compressed reparametrized representation and the one or more decoder models to an edge device that is configured to reconstruct the plurality of model parameters from the compressed reparametrized representation through decompression and employment of the one or more decoder models. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: partitioning, by the one or more computing devices, the plurality of model parameters and the plurality of reparametrized representations into a plurality of groups. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, further comprises: generating, by the one or more computing devices, a plurality of decoder models respectively associated with the plurality of groups, wherein the plurality of decoder models includes the one or more decoder models and each decoder model of the plurality of decoder models is configured to transform the plurality of reparametrized representations included in a corresponding group into the plurality of model parameters included in the corresponding group. ### Claim 9 (depends on 7) The computer-implemented method of claim 7, wherein partitioning the plurality of model parameters and the plurality of reparametrized representations into the plurality of groups comprises: partitioning, by the one or more computing devices, the plurality of model parameters into the plurality of groups based at least in part on a respective parameter type associated with each model parameter, wherein at least some of the respective parameter types comprise the following types: convolutional, fully connected, and bias. ### Claim 10 (depends on 7) The computer-implemented method of claim 7, wherein each group of model parameters is treated as samples from a respective learned distribution. ### Claim 11 (depends on 7) The computer-implemented method of claim 7, further comprising: defining, by the one or more computing devices, the plurality of reparametrized representations based on a fully factorized distribution with parameter sharing within factors of the distribution that correspond to a same group of the plurality of groups. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the loss function comprises a loss term that provides a loss value that is positively correlated to an entropy associated with the plurality of reparametrized representations. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the loss function comprises a loss term that provides a loss value that is positively correlated to a self-information of the plurality of reparametrized representations over one or more probability distributions. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the one or more probability distributions are estimated using one or more non-parametric continuous density functions. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, further comprising: maintaining, by the one or more computing devices, a plurality of continuous surrogates of the reparametrized representations; and modifying, by the one or more computing devices based on the loss function, the plurality of continuous surrogates of the reparametrized representations; after modifying the plurality of continuous surrogates, rounding, by the one or more computing devices, each of the plurality of continuous surrogates to a respective nearest quantization value to obtain a plurality of final reparametrized representations. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein rounding each of the plurality of continuous surrogates to the respective nearest quantization value comprises: rounding, by the one or more computing devices, each of the plurality of continuous surrogates to a respective nearest integer. ### Claim 17 (depends on 1) The computer-implemented method of claim 1, wherein the one or more decoder models comprise one or more linear functions. ### Claim 18 (depends on 1) The computer-implemented method of claim 1, wherein the one or more decoder models comprise one or more non-linear functions. ### Claim 19 One or more non-transitory computer-readable media that store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations for learning a compressed representation of a machine-learned model via a representation within a reparameterization space, the operations comprising: employing the machine-learned model to generate a prediction based on a training input, wherein an uncompressed representation of the machine-learned model is characterized by a plurality of model parameters that are representable, within the reparameterization space, by a plurality of reparametrized representations; modifying at least one data element of a set of data elements based on an evaluation of a loss function that is a function of the prediction, wherein the set of data elements includes at least the plurality of parameterized representations and one or more decoder models that are configured to transform the plurality of reparametrized representations into the plurality of model parameters; after modifying the at least one data element, generating the compressed reparametrized representation of the machine-learned model based on applying one or more entropy coding compression techniques to the plurality of reparametrized representations. ### Claim 20 A computing device comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when the instructions are executed by the one or more processors, cause the computer system to perform operations to learn a compressed representation of a machine-learned model via a representation within a reparameterization space, the operations comprising: employing, by the one or more computing devices, the machine-learned model to generate a prediction based on a training input, wherein an uncompressed representation of the machine-learned model is characterized by a plurality of model parameters that are representable, within the reparameterization space, by a plurality of reparametrized representations; modifying, by the one or more computing devices, at least one data element of a set of data elements based on an evaluation of a loss function that is a function of the prediction, wherein the set of data elements includes at least the plurality of parameterized representations and one or more decoder models that are configured to transform the plurality of reparametrized representations into the plurality of model parameters; after modifying the at least one data element, generating, by the one or more computing devices, the compressed reparametrized representation of the machine-learned model based on applying one or more entropy coding compression techniques to the plurality of reparametrized representations.