# US 12,182,963 B2 · Image processing apparatus and method implementing a neural network to pre-process a CFM image for further process by ISP pipeline > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,963 B2 - Document ID: US12182963B2 - Status: Granted - Kind: B2 - Application number: 17/458,699 - Filed: 2021-08-27 - Published: 2021-12-16 - Granted: 2024-12-31 - Earliest claimed: 2019-02-27 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Nickolay Dmitrievich Egorov; Elena Alexandrovna Alshina; Marat Ravilevich Gilmutdinov; Dmitry Vadimovich Novikov; Anton Igorevich Veselov; Kirill Aleksandrovich Malakhov - Web page: https://patentgrep.com/patent/US12182963B2 ## CPC - H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/70: Denoising; Smoothing - G06T 5/73: Deblurring; Sharpening - G06T 5/90: Dynamic range modification of images or parts thereof - H04N 23/86: for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits - H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control - G06T 2207/10024: Color image - G06T 2207/10144: Varying exposure - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - H04N 25/13: characterised by the spectral characteristics of the filter elements - H04N 2209/045: using mosaic colour filter - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing apparatus processes a color filter mosaic, CFM, image of a scene into a final image of the scene. The image processing apparatus includes processing circuitry configured to implement a neural network. The neural network is configured to process the CFM image into an enhanced CFM image. The processing circuitry is further configured to transform the enhanced CFM image into the final image. ## Claims ### Claim 1 An image processing apparatus for processing an original color filter mosaic (CFM) image of a scene into a final image of the scene, the image processing apparatus comprising processing circuitry configured to: implement a neural network and process the original CFM image into an enhanced CFM image using the neural network, the original CFM image and the enhanced CFM image corresponding to a same CFM, wherein the neural network is implemented as a pre-processing stage prior to and independent of an image signal processing (ISP) pipeline, so that no retraining of the neural network is required in case of modifications of the ISP pipeline, and transform, by applying the ISP pipeline, the enhanced CFM image into the final image. ### Claim 2 (depends on 1) The image processing apparatus of claim 1, wherein the CFM is one of the following: a Bayer filter mosaic, a red green blue emerald (RGBE) filter mosaic, a cyan yellow yellow magenta (CYYM) filter mosaic, a cyan yellow green magenta (CYGM) filter mosaic, or an X-Trans filter mosaic. ### Claim 3 (depends on 2) The image processing apparatus of claim 2, wherein the transforming, by applying the ISP pipeline, the enhanced CFM image into the final image comprises de-mosaicing. ### Claim 4 (depends on 1) The image processing apparatus of claim 1, wherein each or both of the processing the original CFM image into the enhanced CFM image and the transforming, by applying the ISP pipeline, the enhanced CFM image into the final image comprises one or more of the following: denoising, white-balancing, tone mapping, contrast enhancement, or sharpening. ### Claim 5 (depends on 1) The image processing apparatus of claim 1, wherein the image processing apparatus further comprises an image capturing device configured to generate the original CFM image of the scene. ### Claim 6 (depends on 1) The image processing apparatus of claim 1, wherein each of the original CFM image and the enhanced CFM image comprises a plurality of samples correspond to a plurality of color channels, and wherein the pattern of the plurality of samples and the number of the plurality of color channels associated with the original CFM image are the same as those of the enhanced CFM image. ### Claim 7 (depends on 1) The image processing apparatus of claim 1, wherein the neural network remains the same in response to a modification to the ISP pipeline. ### Claim 8 An image processing method for processing an original color filter mosaic (CFM) image of a scene into a final image of the scene, the method comprising: processing the original CFM image into an enhanced CFM image using a neural network, wherein the neural network is implemented as a pre-processing stage prior to and independent of an image signal processing (ISP) pipeline, so that no retraining of the neural network is required in case of modifications of the ISP pipeline; and transforming, by applying the ISP pipeline, the enhanced CFM image into the final image. ### Claim 9 (depends on 8) A non-transitory computer-readable storage medium carrying program code which causes a computer or a processor to perform the method of claim 8 when the program code is executed by the computer or the processor. ### Claim 10 (depends on 8) The method of claim 8, wherein each of the original CFM image and the enhanced CFM image comprises a plurality of samples correspond to a plurality of color channels, and wherein the pattern of the plurality of samples and the number of the plurality of color channels associated with the original CFM image are the same as those of the enhanced CFM image. ### Claim 11 (depends on 8) The method of claim 8, wherein the neural network remains the same in response to a modification to the ISP pipeline.