# US 11,308,592 B2 · Image processing method, image processing apparatus, imaging apparatus, and storage medium, that correct a captured image using a neutral network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,592 B2 - Document ID: US11308592B2 - Status: Granted - Kind: B2 - Application number: 16/583,645 - Filed: 2019-09-26 - Published: 2020-04-09 - Granted: 2022-04-19 - Earliest claimed: 2018-10-04 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Norihito Hiasa - Web page: https://patentgrep.com/patent/US11308592B2 ## CPC - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning - G06T 3/4023: based on decimating pixels or lines of pixels; based on inserting pixels or lines of pixels - G06T 5/20: using local operators - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/73: Deblurring; Sharpening - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/90: Determination of colour characteristics - H04N 23/80: Camera processing pipelines; Components thereof - G06T 2207/10024: Color image - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20212: Image combination - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image processing method configured to correct a captured image using a neural network includes a first step of determining an inversion axis for a partial image or a filter of the neural network according to a position in the captured image of the partial image that is part of the captured image, a second step of determining a positional relationship among pixels of color components in an image input to the neural network corresponding to the inversion axis, and a third step of generating a corrected image obtained by correcting an input image by processing, using the neural network, the input image generated from the captured image based on the positional relationship. ## Claims ### Claim 1 An image processing method of correcting at least part of a captured image using a neural network, the image processing method comprising: a first step of determining, according to a position of a partial image of the captured image, an inversion axis that is a line on the partial image or on a filter of the neural network, the inversion axis being defined by an azimuth; a second step of determining, based on the inversion axis, a positional relationship among pixels of color components in an input image that is generated from the captured image based on the positional relationship and is input to the neural network; a third step of: in a case where the first step determines the inversion axis on the partial image, generating the input image by spatially inverting the partial image with respect to the inversion axis determined by the first step and by adjusting arrangement of color components in the input image so as to correspond to the positional relationship determined by the second step; and in a case where the first step determines the inversion axis on the filter of the neural network, spatially inverting the filter of the neural network with respect to the inversion axis determined by the first step and generating the input image by adjusting the arrangement of color components in the input image so as to correspond to the positional relationship determined by the second step; and a fourth step of generating a corrected image by correcting the generated input image by processing the input image using the neural network. ### Claim 2 (depends on 1) The image processing method according to claim 1, further comprising: a combining step of generating a corrected captured image obtained by correcting the captured image by combining a plurality of corrected images calculated for a plurality of partial images, wherein the plurality of partial images include a first partial image and a second partial image in which the inversion axis different from that of the first partial image is determined in the first step. ### Claim 3 (depends on 2) The image processing method according to claim 2, wherein the neural network is common to the first partial image and the second partial image. ### Claim 4 (depends on 2) The image processing method according to claim 2, wherein the first partial image includes part of the second partial image. ### Claim 5 (depends on 1) The image processing method according to claim 1, wherein the fourth step generates the input image by cutting out part of the captured image in an area determined based on the positional relationship. ### Claim 6 (depends on 1) The image processing method according to claim 1, wherein the fourth step generates the input image by deleting a partial pixel in the partial image or by changing a position of a partial pixel of the partial image based on the positional relationship. ### Claim 7 (depends on 1) The image processing method according to claim 1, further comprising: a step of extracting a plurality of partial images from the captured image, wherein the first step determines an inversion axis for each of a plurality of extracted partial images, based on the largest region as the largest area occupied by the partial images, among a plurality of regions made by dividing the captured image for each angle of integer multiple of 45° around a reference point. ### Claim 8 (depends on 7) The image processing method according to claim 7, wherein the reference point is a point corresponding to an optical axis of an imaging optical system used to generate the captured image. ### Claim 9 (depends on 7) The image processing method according to claim 7, wherein the first step performs the inversion processing based on a position in the captured image of a region including the partial image among the plurality of regions. ### Claim 10 (depends on 1) The image processing method according to claim 1, wherein the second step determines the positional relationship based on a positional relationship among color components in a training image used for learning of the neural network. ### Claim 11 (depends on 1) The image processing method according to claim 1, wherein the corrected image is an image in which an optical degradation in the captured image is corrected. ### Claim 12 An image processing apparatus for correcting at least part of a captured image using a neural network, the image processing apparatus comprising: at least one processor configured to execute a plurality of tasks, including: an inversion determining task that determines, according to a position of a partial image of the captured image, an inversion axis that is a line on the partial image or on a filter of the neural network, the inversion axis being defined by an azimuth; a color determining task that determines, based on the inversion axis, a positional relationship among pixels of color components in an input image that is generated from the captured image based on the positional relationship and is input to the neural network; an image generating task that: in a case where the inversion determining task determines the inversion axis on the partial image, generates the input image by spatially inverting the partial image with respect to the inversion axis determined by the inversion determining task and by adjusting arrangement of color components in the input image so as to correspond to the positional relationship determined by the color determining task; and in a case where the inversion determining task determines the inversion axis on the filter of the neural network, spatially inverts the filter of the neural network with respect to the inversion axis determined by the first step and generates the input image by adjusting the arrangement of color components in the input image so as to correspond to the positional relationship determined by the color determining task; and a corrected image generating task that generates a corrected image by correcting the input image by processing the generated input image using the neural network. ### Claim 13 (depends on 12) The image processing apparatus according to claim 12, further comprising a memory configured to store a filter used in the neural network. ### Claim 14 (depends on 12) An imaging apparatus comprising: an imager configured to capture an object image; and the image processing apparatus according to claim 12. ### Claim 15 An image processing system comprising: a first apparatus; and a second apparatus, wherein the first apparatus includes a transmitter configured to transmit a request for causing the second apparatus to execute processing for correcting a captured image, and wherein the second apparatus includes: a receiver configured to receive the request; at least one processor configured to execute a plurality of tasks, including: an inversion determining task that determines, according to a position of a partial image of the captured image, an inversion axis that is a line on the partial image or on a filter of the neural network, the inversion axis being defined by an azimuth; a color determining task determines, based on the inversion axis, a positional relationship among pixels in color components of an input image that is generated from the captured image based on the positional relationship and is input to the neural network; an image generating task that: in a case where the inversion determining task determines the inversion axis on the partial image, generates the input image by spatially inverting the partial image with respect to the inversion axis determined by the inversion determining task and by adjusting arrangement of color components in the input image so as to correspond to the positional relationship determined by the color determining the task; and in a case where the inversion determining task determines the inversion axis on the filter of the neural network, spatially inverts the filter of the neural network with respect to the inversion axis determined by the first step and generates the input image by adjusting the arrangement of color components in the input image so as to correspond to the positional relationship determined by the color determining task; and a corrected image generating task that generates a corrected image by correcting the input image by processing the generated input image using the neural network. ### Claim 16 A non-transitory computer-readable storage medium storing a program executable by a computer to execute an image processing method of corrrecting at least part of a captured image using a neural network, the image processing method comprising: a first step of determining, according to a position of a partial image of the captured image, an inversion axis that is a line on the partial image or on a filter of the neural network, the inversion axis being defined by an azimuth; a second step of determining, based on the inversion axis, a positional relationship among pixels of color components in an input image that is generated from the captured image based on the positional relationship and is input to the neural network; a third step of: in a case where the first step determines the inversion axis on the partial image, generating the input image by spatially inverting the partial image with respect to the inversion axis determined by the first step and by adjusting arrangement of color components in the input image so as to correspond to the positional relationship determined by the second step; and in a case where the first step determines the inversion axis on the filter of the neural network, spatially inverting the filter of the neural network with respect to the inversion axis determined by the first step and generating the input image by adjusting the arrangement of color components in the input image so as to correspond to the positional relationship determined by the second step; and a fourth step of generating a corrected image by correcting the generated input image by processing the input image using the neural network.