# US 10,771,762 B2 · Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium that correct a parallax image based on a correction value calculated using a captured image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,762 B2 - Document ID: US10771762B2 - Status: Granted - Kind: B2 - Application number: 15/724,471 - Filed: 2017-10-04 - Published: 2018-02-08 - Granted: 2020-09-08 - Earliest claimed: 2015-05-08 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Koichi Fukuda - Web page: https://patentgrep.com/patent/US10771762B2 ## CPC - H04N 13/128: Adjusting depth or disparity - G03B 13/36: Autofocus systems - G03B 35/10: having single camera with stereoscopic-base-defining system - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/50: Depth or shape recovery - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 13/225: using parallax barriers - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - H04N 25/134: {based on three different wavelength filter elements} - H04N 25/61: the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" - H04N 25/704: Pixels specially adapted for focusing, e.g. phase difference pixel sets - G06T 2207/10024: Color image - G06T 2207/10052: Images from lightfield camera - H04N 13/257: Colour aspects - H04N 23/67: Focus control based on electronic image sensor signals - H04N 23/76: by influencing the image signals - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing apparatus includes an acquisition unit that acquires a parallax image generated based on a signal from one of a plurality of photoelectric converters that receive light beams passing through partial pupil regions of an imaging optical system different from each other, and a captured image generated by combining a plurality of signals from the plurality of photoelectric converters. A determination unit determines whether the parallax image contains a defect, and an image processing unit corrects, if the determination unit determines that the parallax image contains the defect, one or more pixel values of the parallax image that include the defect, using the captured image to calculate a correction value for each of the one or more pixels in the parallax image, and replaces the one or more pixels in the parallax image with the corresponding correction value. ## Claims ### Claim 1 An image processing apparatus comprising: (A) a memory storing instructions; (B) one or more processors that execute the instructions stored in the memory; and (C) an image processing circuit that, based on the instructions executed by the one or more processors, is configured to function as: (a) an acquisition unit configured to acquire (i) a parallax image generated based on a signal from one of a plurality of photoelectric converters that receive light beams passing through partial pupil regions of an imaging optical system different from each other, and (ii) a captured image generated by combining a plurality of signals from the plurality of photoelectric converters; (b) a determination unit configured to determine whether the parallax image contains a defect; (c) an image processing unit configured (i) to calculate a correction value by using pixel values of the captured image corresponding to the defect determined by the determination unit, and (ii) to correct one or more pixel values of the parallax image that include the defect with the corresponding correction value; and (d) a storage unit configured to store the parallax image corrected by the image processing unit in a storage medium. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the determination unit determines whether the parallax image contains the defect by comparing the parallax image with the captured image. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the image processing unit corrects the pixel value of the parallax image at a first position that contains the defect, based on a comparison of the pixel value of the parallax image with a pixel value of the captured image at the first position. ### Claim 4 (depends on 3) The image processing apparatus according to claim 3, wherein the image processing unit corrects the pixel value of the parallax image at the first position based on the pixel value of the captured image at the first position, a pixel value of the parallax image at a second position that does not contain a defect, and a pixel value of the captured image at the second position. ### Claim 5 (depends on 4) The image processing apparatus according to claim 4, wherein the second position is a position adjacent to the first position in a predetermined direction. ### Claim 6 (depends on 4) The image processing apparatus according to claim 4, wherein, when a luminance value of one of the pixel value of the parallax image and the pixel value of the captured image at the second position is less than a predetermined luminance value, the image processing unit replaces the pixel value of the parallax image with the predetermined luminance value. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the predetermined luminance value is set to change depending on a number of partial pupil regions. ### Claim 8 (depends on 6) The image processing apparatus according to claim 6, wherein the predetermined luminance value is set to change depending on the first position. ### Claim 9 (depends on 6) The image processing apparatus according to claim 6, wherein the predetermined luminance value is set to change depending on imaging condition information. ### Claim 10 (depends on 9) The image processing apparatus according to claim 9, wherein the imaging condition information includes at least one of an International Organization of Standardization (ISO) speed, an F-number of the imaging optical system, and an exit pupil distance. ### Claim 11 (depends on 3) The image processing apparatus according to claim 3, wherein the memory is configured to store defect information on the first position, and wherein the image processing unit corrects the parallax image based on the stored defect information. ### Claim 12 (depends on 3) The image processing apparatus according to claim 3, further comprising (D) an inspector configured to inspect defect information on the first position, wherein the image processing unit corrects the parallax image based on the defect information. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the parallax image is generated by collecting light receiving signals from a plurality of subpixels included in the one photoelectric converter, of the plurality of photoelectric converters, for each of the partial pupil regions in the imaging optical system, the partial pupil regions being different from each other, and wherein the captured image is generated by collecting light receiving signals from all subpixels included in the plurality of photoelectric converters. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the parallax images include a first parallax image and a second parallax image, which correspond to light beams passing through the partial pupil regions in the imaging optical system, the partial pupil regions being different from each other, wherein the acquisition unit acquires the captured image and the first parallax image, among the parallax images, from an image pickup element including the plurality of photoelectric converters, and wherein the image processing unit is further configured: (iii) to correct the first parallax image, among the parallax images; and (iv) to generate the second parallax image by subtracting the corrected first parallax image from the captured image. ### Claim 15 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit performs refocus process for the captured image based on the corrected parallax image. ### Claim 16 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit performs a light amount correction process for the parallax image based on the captured image. ### Claim 17 (depends on 16) The image processing apparatus according to claim 16, wherein the image processing unit performs the light amount correction process for the parallax image based on the captured image for each color of the parallax image based on the captured image. ### Claim 18 (depends on 16) The image processing apparatus according to claim 16, wherein the image processing unit performs the light amount correction process for the parallax image based on a projection signal of the captured image and a projection signal of the parallax image. ### Claim 19 (depends on 16) The image processing apparatus according to claim 16, wherein the image processing unit corrects the parallax image after the light amount correction process is performed, based on the captured image, after the light amount correction process is performed for the parallax image. ### Claim 20 An image pickup apparatus comprising: (A) an image pickup element including a plurality of arrayed pixels that include a plurality of photoelectric converters that receive light beams passing through partial pupil regions in an imaging optical system, the partial pupil regions being different from each other; (B) a memory storing instructions; (C) one or more processors that execute the instructions stored in the memory; and (D) an image processing circuit that, based on the instructions executed by the one or more processors, is configured to function as: (a) an acquisition unit configured to acquire (i) a parallax image generated based on a signal from one of the plurality of photoelectric converters, and (ii) a captured image generated by combining signals from the plurality of photoelectric converters; (b) a determination unit configured to determine whether the parallax image contains a defect; (c) an image processing unit configured (i) to calculate a correction value by using pixel values of the captured image corresponding to the defect determined by the determination unit, and (ii) to correct one or more pixel values of the parallax image that include the defect with the corresponding correction value; and (d) a storage unit configured to store the parallax image corrected by the image processing unit in a storage medium. ### Claim 21 (depends on 20) The image pickup apparatus according to claim 20, wherein the image pickup element includes the plurality of photoelectric converters for a single micro lens, and the micro lens is two-dimensionally arrayed. ### Claim 22 An image processing method comprising the steps of: acquiring (i) a parallax image generated based on a signal from one of a plurality of photoelectric converters that receive light beams passing through partial pupil regions of an imaging optical system different from each other, and (ii) a captured image generated by combining a plurality of signals from the plurality of photoelectric converters; determining whether the parallax image contains a defect; calculating a correction value by using pixel values of the captured image corresponding to the defect determined by the determining; correcting one or more pixel values of the parallax image that include the defect with the corresponding correction value; and storing the parallax image corrected by the correcting in a storage medium. ### Claim 23 A non-transitory computer-readable storage medium storing a program that causes a computer to execute an image processing method comprising the steps of: acquiring (i) a parallax image generated based on a signal from one of a plurality of photoelectric converters that receive light beams passing through partial pupil regions of an imaging optical system different from each other, and (ii) a captured image generated by combining a plurality of signals from the plurality of photoelectric converters; determining whether the parallax image contains a defect; calculating a correction value by using the pixel values of the captured image corresponding to the defect determined by the determining; correcting one or more pixel values of the parallax image with the corresponding correction value; and storing the parallax image corrected by correcting in a storage medium. ### Claim 24 (depends on 1) The image processing apparatus according to claim 1, wherein the correction value Ac(i, j), for a pixel at a position i, j in the parallax image, is calculated using a pixel value A(i, j) of the parallax image and a pixel value I(i, j) of the captured image, in the following expression: A ⁢ c ⁡ ( j , ⁢ i ) = max ⁡ ( ∑ k = i - 1 i + 1 ⁢ [ A ⁡ ( j - 1 , k ) + A ⁡ ( j + 1 , k ) , A ⁢ ⁢ 0 ] ) max ⁡ ( ∑ k = i - 1 i + 1 ⁢ [ I ⁡ ( j - 1 , k ) + I ⁡ ( j + 1 , k ) , I ⁢ ⁢ 0 ] ) × I ⁡ ( j , ⁢ i ) .