# US 12,225,239 B2 · High-fidelity generative image compression > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,225,239 B2 - Document ID: US12225239B2 - Status: Granted - Kind: B2 - Application number: 18/238,068 - Filed: 2023-08-25 - Published: 2024-03-28 - Granted: 2025-02-11 - Earliest claimed: 2020-11-30 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: George Dan Toderici; Fabian Julius Mentzer; Eirikur Thor Agustsson; Michael Tobias Tschannen - Web page: https://patentgrep.com/patent/US12225239B2 ## CPC - H04N 19/463: by compressing encoding parameters before transmission - H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - H04N 19/124: Quantisation - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an encoder neural network configured to receive a data item and to process the data item to output a compressed representation of the data item. In one aspect, a method includes, for each training data item: processing the data item using the encoder neural network to generate a latent representation of the training data item; processing the latent representation using a hyper-encoder neural network to determine a conditional entropy model; generating a compressed representation of the training data item; processing the compressed representation using a decoder neural network to generate a reconstruction of the training data item; processing the reconstruction of the training data item using a discriminator neural network to generate a discriminator network output; evaluating a first loss function; and determining an update to the current values of the encoder network parameters. ## Claims ### Claim 1 A method performed by one or more computers for training an encoder neural network configured to receive a data item and to process the data item in accordance with current values of a plurality of encoder network parameters to output a compressed representation of the data item, wherein the training comprises, receiving a plurality of training data items, and, for each training data item: processing the training data item using the encoder neural network to generate a latent representation of the training data item; generating a compressed representation of the training data item from the latent representation of the training data item; processing the compressed representation using a decoder neural network to generate a reconstruction of the training data item; processing the reconstruction of the training data item using a discriminator neural network to generate a discriminator network output that specifies a discriminator's classification of the reconstruction of the training data item; evaluating a first loss function that depends on (ii) a reconstruction term measuring a quality of the reconstruction, and (iii) a discriminator term measuring a difference between the discriminator's classification of the reconstruction of the training data item and a ground truth classification of the training data item; and determining an update to the current values of the encoder network parameters based on determining a gradient with respect to the encoder network parameters of the first loss function. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining an update to current values of decoder network parameters of the decoder neural network based on determining a gradient with respect to the encoder network parameters of the first loss function. ### Claim 3 (depends on 1) The method of claim 1, wherein the data item comprises an image or a video frame. ### Claim 4 (depends on 1) The method of claim 1, wherein the encoder neural network and the decoder neural network are each a respective convolutional neural network. ### Claim 5 (depends on 1) The method of claim 1, further comprising determining an update to current values of discriminator network parameters based on determining a gradient with respect to the discriminator network parameters of a second loss function that depends on the difference between the discriminator's classification of the reconstruction of the training data item and a ground truth classification of the training data item. ### Claim 6 (depends on 1) The method of claim 1, wherein the reconstruction term comprises a term that measures a mean squared error between the training data item and the reconstruction of the training data item. ### Claim 7 (depends on 1) The method of claim 1, wherein the reconstruction term comprises a term that measures a perceptual loss between the training data item and the reconstruction of the training data item. ### Claim 8 (depends on 7) The method of claim 7, wherein the perceptual loss is a Learned Perceptual Image Patch Similarity (LPIPS). ### Claim 9 (depends on 1) The method of claim 1, wherein generating a compressed representation of the training data item from the latent representation of the training data item comprises: quantizing the latent representation of the training data item generated by the encoder neural network. ### Claim 10 A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an encoder neural network configured to receive a data item and to process the data item in accordance with current values of a plurality of encoder network parameters to output a compressed representation of the data item, wherein the training comprises, receiving a plurality of training data items, and, for each training data item: processing the training data item using the encoder neural network to generate a latent representation of the training data item; generating a compressed representation of the training data item from the latent representation of the training data item; processing the compressed representation using a decoder neural network to generate a reconstruction of the training data item; processing the reconstruction of the training data item using a discriminator neural network to generate a discriminator network output that specifies a discriminator's classification of the reconstruction of the training data item; evaluating a first loss function that depends on (ii) a reconstruction term measuring a quality of the reconstruction, and (iii) a discriminator term measuring a difference between the discriminator's classification of the reconstruction of the training data item and a ground truth classification of the training data item; and determining an update to the current values of the encoder network parameters based on determining a gradient with respect to the encoder network parameters of the first loss function. ### Claim 11 (depends on 10) The system of claim 10, further comprising determining an update to current values of decoder network parameters of the decoder neural network based on determining a gradient with respect to the encoder network parameters of the first loss function. ### Claim 12 (depends on 10) The system of claim 10, wherein the data item comprises an image or a video frame. ### Claim 13 (depends on 10) The system of claim 10, wherein the encoder neural network and the decoder neural network are each a respective convolutional neural network. ### Claim 14 (depends on 10) The system of claim 10, further comprising determining an update to current values of discriminator network parameters based on determining a gradient with respect to the discriminator network parameters of a second loss function that depends on the difference between the discriminator's classification of the reconstruction of the training data item and a ground truth classification of the training data item. ### Claim 15 (depends on 10) The system of claim 10, wherein the reconstruction term comprises a term that measures a mean squared error between the training data item and the reconstruction of the training data item. ### Claim 16 (depends on 10) The system of claim 10, wherein the reconstruction term comprises a term that measures a perceptual loss between the training data item and the reconstruction of the training data item. ### Claim 17 (depends on 16) The system of claim 16, wherein the perceptual loss is a Learned Perceptual Image Patch Similarity (LPIPS). ### Claim 18 (depends on 10) The system of claim 10, wherein generating a compressed representation of the training data item from the latent representation of the training data item comprises: quantizing the latent representation of the training data item generated by the encoder neural network. ### Claim 19 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an encoder neural network configured to receive a data item and to process the data item in accordance with current values of a plurality of encoder network parameters to output a compressed representation of the data item, wherein the training comprises, receiving a plurality of training data items, and, for each training data item: processing the training data item using the encoder neural network to generate a latent representation of the training data item; processing the latent representation using a hyper-encoder neural network to determine a conditional entropy model; generating a compressed representation of the training data item including entropy encoding the latent representation of the training data item in accordance with the conditional entropy model; processing the compressed representation using a decoder neural network to generate a reconstruction of the training data item; processing the reconstruction of the training data item using a discriminator neural network to generate a discriminator network output that specifies a discriminator's classification of the reconstruction of the training data item; evaluating a first loss function that depends on (i) a first term measuring a compression rate of the conditional entropy model, (ii) a second term measuring a quality of the reconstruction, and (iii) a third term measuring a difference between the discriminator's classification of the reconstruction of the training data item and a ground truth classification of the training data item; and determining an update to the current values of the encoder network parameters based on determining a gradient with respect to the encoder network parameters of the first loss function. ### Claim 20 (depends on 19) The computer-readable storage media of claim 19, wherein generating a compressed representation of the training data item from the latent representation of the training data item comprises: quantizing the latent representation of the training data item generated by the encoder neural network.