# US 11,445,222 B1 · Preprocessing image data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,222 B1 - Document ID: US11445222B1 - Status: Granted - Kind: B1 - Application number: 17/037,339 - Filed: 2020-09-29 - Granted: 2022-09-13 - Earliest claimed: 2019-09-30 - Assignees: Isize Limited - Applicants: iSize Limited - Inventors: Ioannis Andreopoulos; Aaron Chadha - Web page: https://patentgrep.com/patent/US11445222B1 ## CPC - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06V 10/20: Image preprocessing - G06V 10/82: using neural networks - H04N 19/124: Quantisation - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/179: the unit being a scene or a shot - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - H04N 19/65: using error resilience - H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) - H04L 65/65: Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP] - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Certain aspects of the present disclosure provide techniques for preprocessing, prior to encoding with an external encoder, image data using a preprocessing network comprising a set of inter-connected weights is provided. At the preprocessing network, image data from one or more images is received. The image data is processed using the preprocessing network to generate an output pixel representation for encoding with the external encoder. The weights of the preprocessing network are trained to optimize a combination of at least one quality score indicative of the quality of the output pixel representation and a rate score indicative of the bits required by the external encoder to encode the output pixel representation. ## Claims ### Claim 1 A computer-implemented method of preprocessing, prior to encoding using an external encoder, image data using a preprocessing network comprising a set of inter-connected weights, the method comprising: receiving, at the preprocessing network, image data from one or more images; and processing the image data using the preprocessing network to generate an output pixel representation for encoding with the external encoder, wherein the set of inter-connected weights of the preprocessing network are trained to optimize a combination of: at least one quality score indicative of a quality of the output pixel representation; and a rate score indicative of a number of bits required by the external encoder to encode the output pixel representation, and wherein, during an initial setup or training phase, the at least one quality score is optimized in a direction of improved visual quality or reconstruction, and the rate score is optimized in a direction of lower rate. ### Claim 2 (depends on 1) The method according to claim 1, wherein the at least one quality score is indicative of signal distortion in the output pixel representation. ### Claim 3 (depends on 1) The method according to claim 1, wherein the at least one quality score is indicative of loss of perceptual or aesthetic quality in the output pixel representation. ### Claim 4 (depends on 1) The method according to claim 1, wherein a resolution of the output pixel representation is increased or decreased in accordance with an upscaling or downscaling ratio. ### Claim 5 (depends on 1) The method according to claim 1, further comprising the step of corrupting the output pixel representation by applying one or more mathematically differentiable functions and an approximation, wherein the output pixel representation is corrupted so as to approximate the corruption expected from a block-based transform and quantization used in the external encoder, and/or to approximate the corruption expected from a transform and quantization of errors computed from a block-based temporal prediction process used in the external encoder. ### Claim 6 (depends on 1) The method according to claim 1, further comprising the step of resizing the output pixel representation to a resolution of the image data using a linear or non-linear filter configured during the initial setup or training phase. ### Claim 7 (depends on 1) The method according to claim 1, wherein the least one quality score and the rate score are optimized according to a linear or non-linear optimization method that adjusts the set of inter-connected weights of the preprocessing network and/or adjusts a type of architecture used to interconnect the set of inter-connected weights of the preprocessing network. ### Claim 8 (depends on 1) The method according to claim 1, further comprising the step of encoding the output pixel representation with the external encoder. ### Claim 9 (depends on 1) The method according to claim 1, wherein the external encoder is an ISO JPEG or ISO MPEG standard encoder, or an AOMedia encoder. ### Claim 10 (depends on 1) The method according to claim 1, further comprising filtering the output pixel representation using a linear filter, the linear filter comprising a blur or edge-enhancement filter. ### Claim 11 (depends on 1) The method according to claim 1, wherein the at least one quality score includes one or more of the following: peak-signal-to-noise ratio, structural similarity index metric (SSIM), multiscale quality metrics, detail loss metric or multiscale SSIM, metrics based on multiple quality scores and data-driven learning and training, video multi-method assessment fusion (VMAF), or aesthetic quality metrics. ### Claim 12 (depends on 1) The method according to claim 1, wherein the at least one quality score and the rate score are combined with linear or non-linear weights, and wherein the linear or non-linear weights are trained based on back-propagation and gradient descent methods with representative training data. ### Claim 13 A computing device comprising: a memory comprising computer-executable instructions; a processor configured to execute the computer-executable instructions and cause the computing device to: receive, at a preprocessing network, image data from one or more images; and process the image data using the preprocessing network to generate an output pixel representation for encoding with an external encoder, wherein a set of inter-connected weights of the preprocessing network are trained to optimize a combination of: at least one quality score indicative of a quality of the output pixel representation; and a rate score indicative of a number of bits required by the external encoder to encode the output pixel representation, and wherein, during an initial setup or training phase, the at least one quality score is optimized in a direction of improved visual quality or reconstruction, and the rate score is optimized in a direction of lower rate. ### Claim 14 A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to perform a method, the method comprising: receiving, at a preprocessing network, image data from one or more images; and processing the image data using the preprocessing network to generate an output pixel representation for encoding with an external encoder, wherein a set of inter-connected weights of the preprocessing network are trained to optimize a combination of: at least one quality score indicative of a quality of the output pixel representation; and a rate score indicative of a number of bits required by the external encoder to encode the output pixel representation, and wherein, during an initial setup or training phase, the at least one quality score is optimized in a direction of improved visual quality or reconstruction, and the rate score is optimized in a direction of lower rate. ### Claim 15 (depends on 13) The computing device according to claim 13, wherein the at least one quality score is indicative of signal distortion in the output pixel representation. ### Claim 16 (depends on 13) The computing device according to claim 13, wherein the at least one quality score is indicative of loss of perceptual or aesthetic quality in the output pixel representation. ### Claim 17 (depends on 13) The computing device according to claim 13, wherein a resolution of the output pixel representation is increased or decreased in accordance with an upscaling or downscaling ratio. ### Claim 18 (depends on 14) The non-transitory computer-readable medium according to claim 14, wherein the at least one quality score is indicative of signal distortion in the output pixel representation. ### Claim 19 (depends on 14) The non-transitory computer-readable medium according to claim 14, wherein the at least one quality score is indicative of loss of perceptual or aesthetic quality in the output pixel representation. ### Claim 20 (depends on 14) The non-transitory computer-readable medium according to claim 14, wherein a resolution of the output pixel representation is increased or decreased in accordance with an upscaling or downscaling ratio.