Method and apparatus for multi-learning rates of substitution in neural image compression

An apparatus for substitutional end-to-end (E2E) neural image compression (NIC) using a neural network, the apparatus comprising:

• at least one memory configured to store program code;
• and
• at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
• receiving code configured to cause at least one processor to receive an input image to an E2E NIC framework;
• splitting code configured to cause at least one processor to split the input image into one or more blocks;
• first performing code configured to cause the at least one processor to perform an encoding mapping, for each of the one or more blocks, by mapping the input image to a first bitstream having a first length;
• second performing code configured to cause the at least one processor to perform a decoding mapping, for each of the one or more blocks, by mapping the first bitstream back to an original space with a first distortion loss;
• first determining code configured to cause at least one processor to determine a substitute image from the original space, based on a training model of the E2E NIC framework;
• first encoding code configured to cause at least one processor to encode the substitute image to generate a second bitstream;
• and
• first mapping code configured to cause at least one processor to map the substitute image to the second bitstream to generate a compressed representation,
• wherein the training model of the E2E NIC framework is trained based on a learning rate of the input image, a quantity of updates to the input image, and a second distortion loss,
• wherein a plurality of substitute images are determined based on learning rates that are selected based on characteristics of the input image, and
• wherein the substitute image is determined by performing an optimization process of the training model of the E2E NIC framework, comprising:
• adjusting code configured to cause at least one processor to adjust RGB variance of the split blocks to generate substitute block representations;
• and
• selecting code configured to cause at least one processor to select the RGB variance with a least distortion loss between the split blocks and the substitute block representations to use as the substitute block.

A non-transitory computer readable medium storing instructions that, when executed by at least one processor for substitutional end-to-end (E2E) neural image compression (NIC), cause the at least one processor to:

• receive an input image to an E2E NIC framework;
• split the input image into one or more blocks;
• perform an encoding mapping, for each of the one or more blocks, by mapping the input image to a first bitstream having a first length;
• perform a decoding mapping, for each of the one or more blocks, by mapping the first bitstream back to an original space with a first distortion loss;
• determine a substitute image from the original space, based on a training model of the E2E NIC framework;
• encode the substitute image to generate a second bitstream;
• and
• map the substitute image to the second bitstream to generate a compressed representation,
• wherein the training model of the E2E NIC framework is trained based on a learning rate of the input image, a quantity of updates to the input image, and a second distortion loss,
• wherein a plurality of substitute images are determined based on learning rates that are selected based on characteristics of the input image, and
• wherein the instructions, when executed by at least one processor, further cause the at least one processor to performing an optimization process of the training model of the E2E NIC framework, comprising:
• adjust RGB variance of the split blocks to generate substitute block representations;
• and
• select the RGB variance with a least distortion loss between the split blocks and the substitute block representations to use as the substitute block.