# US 11,909,975 B2 · Dependent scalar quantization with substitution in neural image compression > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,909,975 B2 - Document ID: US11909975B2 - Status: Granted - Kind: B2 - Application number: 17/825,562 - Filed: 2022-05-26 - Published: 2022-12-22 - Granted: 2024-02-20 - Earliest claimed: 2021-06-18 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Sheng Lin; Wei Jiang; Siyu Qi - Web page: https://patentgrep.com/patent/US11909975B2 ## CPC - H04N 19/124: Quantisation - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/13: Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC] - H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) - H04N 19/172: the region being a picture, frame or field - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Method, apparatus, and non-transitory storage medium for end-to-end neural image compression using dependent scalar quantization with substitution, including receiving an input image; determining a substitute image based on the input image using a neural network based substitute feature generator; compressing the substitute image; quantizing the compressed substitute image to obtain a quantized representation of the input image with higher compression performance by using a first dependent scalar quantizer; and entropy encoding the substitute image using a neural network based encoder to generate a compressed representation of the quantized representation. ## Claims ### Claim 1 A method of end-to-end neural image compression using dependent scalar quantization with substitution, the method being executed by one or more processors, and the method comprising: receiving an input image; determining a substitute image based on the input image using a neural network based substitute feature generator; compressing the substitute image; quantizing the compressed substitute image to obtain a quantized representation of the input image with higher compression performance by using a first dependent scalar quantizer; and entropy encoding the substitute image using a neural network based encoder to generate a compressed representation of the quantized representation, wherein a compression quality loss associated with the compressed substitute image is less than a compression quality loss associated with a compressed input image, and wherein one or more neural network models used in the method may be jointly optimized to minimize an overall rate-distortion loss. ### Claim 2 (depends on 1) The method of claim 1, wherein the method further comprises: receiving the compressed representation; decoding the compressed representation using entropy decoding; and generating a reconstructed image based on the decoded compressed representation using a neural network based decoder. ### Claim 3 (depends on 2) The method of claim 2, wherein a best compressed representation is a global minimum of trade-off between a rate-distortion loss between the input image and the reconstructed image and a bit consumption rate of the compressed representation. ### Claim 4 (depends on 1) The method of claim 1, wherein the generating the compressed representation comprises: hyper-encoding the compressed substitute image using a hyper prior neural network based encoder, generating a hyper compressed representation of the hyper-encoded compressed substitute image using a second dependent scalar quantizer and entropy coding; hyper-decoding the hyper compressed representation to generate a hyper reconstructed image using a hyper prior neural network based decoder, training a contextual neural network model based on the hyper reconstructed image and the quantized representation; and generating the compressed representation of the quantized representation using entropy coding and the contextual neural network model. ### Claim 5 (depends on 1) The method of claim 1, wherein the quantizing comprises a first quantizer, a second quantizer, and a state machine, wherein the state machine enables switching between the first quantizer and the second quantizer. ### Claim 6 (depends on 1) The method of claim 1, wherein the neural image compression comprises a pre-trained model, and wherein one or more model weights associated with the pre-trained model are fixed. ### Claim 7 (depends on 6) The method of claim 6, wherein the pre-trained model is fine-tuned using the first dependent scalar quantizer. ### Claim 8 (depends on 1) The method of claim 1, wherein the neural image compression comprises a trained model, and wherein training the trained model comprises: initializing a learning rate for the trained model; adjusting the learning rate for the trained model a threshold number of times, wherein the adjusting is based on image properties of one or more training images; and terminating the training based on at least one of: determining that a difference in the learning rate between consecutive iterations is below a learning threshold, determining that an output loss of a loss function is consistent for a first number of iterations, or determining that the learning rate has been adjusted for a maximum number of iterations. ### Claim 9 (depends on 8) The method of claim 8, wherein the adjusting the learning rate is inversely based on the output loss of the loss function. ### Claim 10 An apparatus for end-to-end neural image compression using dependent scalar quantization with substitution, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: first receiving code configured to cause the at least one processor to receive an input image; first determining code configured to cause the at least one processor to determine a substitute image based on the input image using a neural network based substitute feature generator; compressing code configured to cause the at least one processor to compress the substitute image; quantizing code configured to cause the at least one processor to quantize the compressed substitute image to obtain a quantized representation of the input image with higher compression performance by using a first dependent scalar quantizer, and first generating code configured to cause the at least one processor to entropy encode the substitute image using a neural network based encoder to generate a compressed representation of the quantized representation, wherein a compression quality loss associated with the compressed substitute image is less than a compression quality loss associated with a compressed input image, and wherein one or more neural network models used in the method may be jointly optimized to minimize an overall rate-distortion loss. ### Claim 11 (depends on 10) The apparatus of claim 10, the program code further comprising second receiving code configured to cause the at least one processor to receive the compressed representation; decoding code configured to cause the at least one processor to decode the compressed representation using entropy decoding; and second generating code configured to cause the at least one processor to generate a reconstructed image based on the decoded compressed representation using a neural network based decoder. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein a best compressed representation is a global minimum of trade-off between a rate-distortion loss between the input image and the reconstructed image and a bit consumption rate of the compressed representation. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein the quantizing comprises a first quantizer, a second quantizer, and a state machine, wherein the state machine enables switching between the first quantizer and the second quantizer. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the neural image compression comprises a pre-trained model, and wherein one or more model weights associated with the pre-trained model are fixed. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the pre-trained model is fine-tuned using the first dependent scalar quantizer. ### Claim 16 A non-transitory computer readable medium storing instructions that, when executed by at least one processor for end-to-end neural image compression using dependent scalar quantization with substitution, cause the at least one processor to: receive an input image; determine a substitute image based on the input image using a neural network based substitute feature generator; compress the substitute image; quantize the compressed substitute image to obtain a quantized representation of the input image with higher compression performance by using a first dependent scalar quantizer, and entropy encode the substitute image using a neural network based encoder to generate a compressed representation of the quantized representation, wherein a compression quality loss associated with the compressed substitute image is less than a compression quality loss associated with a compressed input image, and wherein one or more neural network model used in the method may be jointly optimized to minimize an overall rate-distortion loss. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: receive the compressed representation; decode the compressed representation using entropy decoding; and generate a reconstructed image based on the decoded compressed representation using a neural network based decoder. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein a best compressed representation is a global minimum of trade-off between a rate-distortion loss between the input image and the reconstructed image and a bit consumption rate of the compressed representation. ### Claim 19 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the dependent scalar quantization comprises a first quantizer, a second quantizer, and a state machine, wherein the state machine enables switching between the first quantizer and the second quantizer. ### Claim 20 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the neural image compression comprises a pre-trained model, and wherein one or more model weights associated with the pre-trained model are fixed.