# US 11,115,601 B2 · Image processing apparatus, imaging apparatus, and image processing method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,601 B2 - Document ID: US11115601B2 - Status: Granted - Kind: B2 - Application number: 16/429,296 - Filed: 2019-06-03 - Published: 2019-12-19 - Granted: 2021-09-07 - Earliest claimed: 2018-06-13 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Yasushi Iwakura - Web page: https://patentgrep.com/patent/US11115601B2 ## CPC - H04N 23/71: Circuitry for evaluating the brightness variation - G06T 5/50: using two or more images, e.g. averaging or subtraction - H04N 23/11: for generating image signals from visible and infrared light wavelengths - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - G06T 2207/10016: Video; Image sequence - G06T 2207/10048: Infrared image - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20212: Image combination - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image processing apparatus including a first acquisition unit for acquiring a visible-light image signal representing a visible-ray image which comes into focus at a first distance and an infrared image signal representing an infrared-ray image which comes into focus at a second distance that is shorter than the first distance and an infrared-ray image which comes into focus at a third distance that is longer than the first distance; a second acquisition unit for acquiring first brightness information, hue information, and saturation information from the visible-light image signal and at least second brightness information from the infrared image signal; a third acquisition unit for acquiring third brightness information based on edge information obtained from the first brightness information and edge information obtained from the second brightness information; and a generation unit for generating a second visible-light image using the third brightness information, the hue information, and the saturation information. ## Claims ### Claim 1 An image processing apparatus comprising a processor configured to operate as: a first acquisition unit configured to acquire (a) a visible light image signal representing a visible-ray image which comes into focus at a first distance and (b) an infrared image signal representing (b1) an infrared-ray image which comes into focus at a second distance that is shorter than the first distance and (b2) an infrared-ray image which comes into focus at a third distance that is longer than the first distance; a second acquisition unit configured to acquire first brightness information, hue information, and saturation information from the visible light image signal and at least second brightness information from the infrared image signal; a third acquisition unit configured to acquire third brightness information based on edge information obtained from the first brightness information and edge information obtained from the second brightness information; and a generation unit configured to generate a second visible light image using the third brightness information, the hue information, and the saturation information. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the first acquisition unit is further configured to acquire, as the infrared image signal, a first infrared image signal representing an infrared-ray image which comes into focus at the second distance and a second infrared image signal representing an infrared-ray image which comes into focus at the third distance, and wherein the second acquisition unit is configured to extract, as the second brightness information, first infrared brightness information obtained from the first infrared image signal and second infrared brightness information obtained from the second infrared image signal. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the third acquisition unit is configured to determine, as the third brightness information, brightness information with a greatest edge strength among the first brightness information, the first infrared brightness information, and the second infrared brightness information. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the first acquisition unit is configured to acquire, as the infrared image signal, an image signal representing an image in which an infrared-ray image which comes into focus at the second distance and an infrared-ray image which comes into focus at the third distance are superimposed on one another. ### Claim 5 (depends on 4) The image processing apparatus according to claim 4, wherein the third acquisition unit is configured to determine, as the third brightness information, brightness information with a greatest edge strength among the first brightness information and the second brightness information. ### Claim 6 (depends on 4) The image processing apparatus according to claim 4, wherein the third acquisition unit is configured to determine the third brightness information based on the first brightness information, the second brightness information, and the saturation information. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the third acquisition unit is configured to determine the first brightness information as the third brightness information in a case where the saturation information is equal to or smaller than a first threshold, determine the second brightness information as the third brightness information in a case where the saturation information is larger than a second threshold that is larger than the first threshold, and determine the brightness information with a greatest edge strength between the first brightness information and the second brightness information as the third brightness information in a case where the saturation information is larger than the first threshold and smaller than the second threshold. ### Claim 8 (depends on 6) The image processing apparatus according to claim 6, wherein the third acquisition unit is configured to compare an edge strength obtained from the first brightness information with a value calculated by weighting and an edge strength obtained from the second brightness information by a weight based on the saturation information, and determine the third brightness information based on a result of the comparison. ### Claim 9 An image processing method performed by an image processing apparatus, the image processing method comprising: a first acquisition step of extracting first brightness information, hue information, and saturation information from a visible light image signal representing a visible-ray image which comes into focus at a first distance, and acquiring at least second brightness information from an infrared image signal representing an infrared-ray image which comes into focus at a second distance that is shorter than the first distance and an infrared-ray image which comes into focus at a third distance that is longer than the first distance; a second acquisition step of acquiring third brightness information based on edge information obtained from the first brightness information and edge information obtained from the second brightness information; and a generation step of generating a second visible light image using the third brightness information, the hue information, and the saturation information. ### Claim 10 (depends on 9) The image processing method according to claim 9, wherein the infrared image signal includes a first infrared image signal representing an infrared-ray image which comes into focus at the second distance and a second infrared image signal representing an infrared-ray image which comes into focus at the third distance, and wherein, in the first acquisition step, first infrared brightness information obtained from the first infrared image signal and second infrared brightness information obtained from the second infrared image signal are respectively extracted as the second brightness information. ### Claim 11 (depends on 10) The image processing method according to claim 10, wherein, in the second acquisition step, brightness information with a greatest edge strength among the first brightness information, the first infrared brightness information, and the second infrared brightness information is determined as the third brightness information. ### Claim 12 (depends on 9) The image processing method according to claim 9, wherein the infrared image signal is an image signal representing an image in which an image in an infrared light range which comes into focus at the second distance and an image in an infrared light range which comes into focus at the third distance are superimposed on one another. ### Claim 13 (depends on 12) The image processing method according to claim 12, wherein, in the second acquisition step, brightness information with a greatest edge strength among the first brightness information and the second brightness information is determined as the third brightness information. ### Claim 14 (depends on 12) The image processing method according to claim 12, wherein, in the second acquisition step, the third brightness information is determined based on the first brightness information, the second brightness information, and the saturation information. ### Claim 15 (depends on 14) The image processing method according to claim 14, wherein in the second acquisition step, the first brightness information is determined as the third brightness information when the saturation information is equal to or smaller than a first threshold, the second brightness information is determined as the third brightness information when the saturation information is larger than a second threshold that is larger than the first threshold, and the brightness information with a greatest edge strength between the first brightness information and the second brightness information is determined as the third brightness information in a case where the saturation information is larger than the first threshold and smaller than the second threshold. ### Claim 16 (depends on 14) The image processing method according to claim 14, wherein, in the second acquisition step, an edge strength obtained from the first brightness information is compared with a value calculated by weighting an edge strength obtained from the second brightness information by a weight based on the saturation information, and the third brightness information is determined based on a result of the comparison. ### Claim 17 (depends on 9) A non-transitory computer-readable medium which stores a program that causes a computer to execute the method according to claim 9. ### Claim 18 (depends on 1) An imaging apparatus comprising: the image processing apparatus according to claim 1, wherein the image processing apparatus generates the second visible light image from a visible light image signal and an infrared image signal. ### Claim 19 (depends on 1) A moving body comprising: the image processing apparatus according to claim 1; a moving apparatus; a processing apparatus configured to acquire information from a signal output from the image processing apparatus; and a control apparatus configured to control the moving apparatus based on the information.