# US 2021/0243344 A1 · Dual sensor imaging system and depth map calculation method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0243344 A1 - Document ID: US20210243344A1 - Status: Published application - Kind: A1 - Application number: 17/214,946 - Filed: 2021-03-29 - Published: 2021-08-05 - Earliest claimed: 2020-09-04 - Assignees: Altek Semiconductor Corp. - Applicants: Altek Semiconductor Corp. - Inventors: Shih-Yuan Peng; Shu-Chun Cheng; Hsu-Lien Huang; Yun-Chin Li; Kuo-Ming Lai - Web page: https://patentgrep.com/patent/US20210243344A1 ## CPC - G06T 7/593: from stereo images - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - G01B 11/24: for measuring contours or curvatures - G06T 7/44: using image operators, e.g. filters, edge density metrics or local histograms - G06T 7/50: Depth or shape recovery - H04N 23/20: for generating image signals from infrared radiation only - H04N 23/71: Circuitry for evaluating the brightness variation - H04N 23/73: by influencing the exposure time - H04N 25/131: including elements passing infrared wavelengths - H04N 25/60: Noise processing, e.g. detecting, correcting, reducing or removing noise - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T 2207/10144: Varying exposure - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS ## Abstract A dual sensor imaging system and a depth map calculation method thereof are provided. The dual sensor imaging system includes at least one color sensor, at least one infrared ray (IR) sensor, a storage device, and a processor. The processor is configured to load and execute a computer program stored in the storage device to: control the color sensor and the IR sensor to respectively capture multiple color images and multiple IR images by adopting multiple exposure conditions suitable for an imaging scene, adaptively select a combination of the color image and the IR image that are comparable to each other from the color images and the IR images; and calculate a depth map of the imaging scene by using the selected color image and IR image. ## Claims ### Claim 1 A dual sensor imaging system, comprising: at least one color sensor; at least one infrared ray (IR) sensor; a storage device, storing a computer program; and a processer, coupled to the at least one color sensor, the at least one IR sensor, and the storage device, and configured to load and execute the computer program to: control the at least one color sensor and the at least one IR sensor to respectively capture a plurality of color images and a plurality of IR images by adopting a plurality of exposure conditions suitable for an imaging scene; adaptively select a combination of the color image and the IR image that are comparable to each other from the color images and the IR images; and calculate a depth map of the imaging scene by using the selected color image and IR image. ### Claim 2 (depends on 1) The dual sensor imaging system according to claim 1, wherein the processor: selects a color image that can reveal color details of the imaging scene from the color images as a reference image; identifies at least one defect area lacking texture details in the reference image; and selects an IR image that can reveal the texture details of the defect area from the IR images to be used as a combination for comparison with the reference image. ### Claim 3 (depends on 2) The dual sensor imaging system according to claim 2, wherein the processor: uses the color image to execute auto focus, so as to obtain a focal length of an object captured in the imaging scene, and determine a color image that can reveal the most color details of the object as the reference image according to the focal length. ### Claim 4 (depends on 1) The dual sensor imaging system according to claim 1, wherein the processor: calculates image histograms of each of the color images and each of the IR images; and selects one of the color images or one of the IR images as a reference, and compares the image histograms of each of the color images and each of the IR images to determine the combination of the color image and the IR image that are comparable to each other. ### Claim 5 (depends on 4) The dual sensor imaging system according to claim 4, wherein the processor: selects one of the color images as a reference image, selects one of the IR images to be compared with the reference image, and determines whether brightness of the selected IR image is higher than brightness of the reference image according to the image histograms, wherein when yes, an IR image with an exposure time shorter than an exposure time of the selected IR image is selected from the captured IR images, or the at least one IR sensor is controlled to capture an IR image by adopting an exposure time shorter than the exposure time of the selected IR image to be used as a combination for comparison with the reference image; and when no, an IR image with an exposure time longer than the exposure time of the selected IR image is selected from the captured IR images, or the at least one IR sensor is controlled to capture an IR image by adopting an exposure time longer than the exposure time of the selected IR image to be used as the combination for comparison with the reference image. ### Claim 6 (depends on 4) The dual sensor imaging system according to claim 4, wherein the processor: selects one of the IR images as a reference image, selects one of the color images to be compared with the reference image, and determines whether brightness of the selected color image is higher than brightness of the reference image according to the image histograms, wherein when yes, a color image with an exposure time shorter than an exposure time of the selected color image is selected from the captured color images, or the at least one color sensor is controlled to capture a color image by adopting an exposure time shorter than the exposure time of the selected color image to be used as a combination for comparison with the reference image; and when no, a color image with an exposure time longer than the exposure time of the selected color image is selected from the captured color images, or the at least one color sensor is controlled to capture a color image by adopting an exposure time longer than the exposure time of the selected color image to be used as the combination for comparison with the reference image. ### Claim 7 (depends on 1) The dual sensor imaging system according to claim 1, wherein the processor: detects at least one occlusion lacking color details or texture details in the selected color image; and executes a hole filling algorithm to determine a depth value of the at least one occlusion according to texture details around each of the at least one occlusion provided by the selected IR image and depth values of a plurality of pixels around each of the at least one occlusion. ### Claim 8 (depends on 7) The dual sensor imaging system according to claim 7, wherein the dual sensor imaging system further comprises an IR projector, and the processor further: controls the IR projector to project an infrared ray to the imaging scene, and controls the at least one IR sensor to capture an IR image of the imaging scene when the at least one occlusion is detected; and determines the depth value of the at least one occlusion according to the texture details around each of the at least one occlusion provided by the captured IR image and the depth values of the pixels around each of the at least one occlusion. ### Claim 9 (depends on 1) The dual sensor imaging system according to claim 1, wherein the processor: captures a plurality of feature points with strong features in the selected color image and IR image; and calculates the depth map of the imaging scene according to positions of the corresponding feature points in the color image and the IR image. ### Claim 10 (depends on 1) The dual sensor imaging system according to claim 1, wherein the processor: controls at least one of the least one color sensor and the at least one IR sensor to capture at least one standard image of the imaging scene by adopting a standard exposure condition, and uses the at least one standard image to identify the imaging scene. ### Claim 11 A depth map calculation method of a dual sensor imaging system, wherein the dual sensor imaging system comprises at least one color sensor, at least one IR sensor, and a processor, the depth map calculation method comprising: controlling the at least one color sensor and the at least one IR sensor to respectively capture a plurality of color images and a plurality of IR images by adopting a plurality of exposure conditions suitable for an imaging scene by the processor; adaptively selecting a combination of the color image and the IR image that are comparable to each other from the color images and the IR images by the processor; and calculating a depth map of the imaging scene by using the selected color image and IR image by the processor. ### Claim 12 (depends on 11) The depth map calculation method according to claim 11, wherein the step of adaptively selecting the combination of the color image and the IR image that are comparable to each other from the color images and the IR images comprises: selecting a color image that can reveal color details of the imaging scene from the color images as a reference image; identifying at least one defect area lacking texture details in the reference image; and selecting an IR image that can reveal the texture details of the defect area from the IR images. ### Claim 13 (depends on 12) The depth map calculation method according to claim 12, wherein the step of selecting the color image that can reveal the color details of the imaging scene from the color images as the reference image comprises: using the color image to execute auto focus, so as to obtain a focal length of an object captured in the imaging scene, and determine a color image that can reveal the most color details of the object as the reference image according to the focal length. ### Claim 14 (depends on 11) The depth map calculation method according to claim 11, wherein the step of adaptively selecting the combination of the color image and the IR image that are comparable to each other from the color images and the IR images comprises calculating image histograms of each of the color images and each of the IR images; and selecting one of the color images or one of the IR images as a reference, and comparing the image histograms of each of the color images and each of the IR images to determine the combination of the color image and the IR image that are comparable to each other. ### Claim 15 (depends on 14) The depth map calculation method according to claim 14, wherein the step of comparing the image histograms of each of the color images and each of the IR images to determine the combination of the color image and the IR image that are comparable to each other comprises: selecting one of the color images as a reference image, selecting one of the IR images to be compared with the reference image, and determining whether brightness of the selected IR image is higher than brightness of the reference image according to the image histograms, wherein when yes, an IR image with an exposure time shorter than an exposure time of the selected IR image is selected from the captured IR images, or the at least one IR sensor is controlled to capture an IR image by adopting an exposure time shorter than the exposure time of the selected IR image to be used as a combination for comparison with the reference image; and when no, an IR image with an exposure time longer than an exposure time of the selected IR image is selected from the captured IR images, or the at least one IR sensor is controlled to capture an IR image by adopting an exposure time longer than the exposure time of the selected IR image to be used as the combination for comparison with the reference image. ### Claim 16 (depends on 14) The depth map calculation method according to claim 14, wherein the step of comparing the image histograms of each of the color images and each of the IR images to determine the combination of the color image and the IR image that are comparable to each other comprises: selecting one of the IR images as a reference image, selecting one of the color images to be compared with the reference image, and determining whether brightness of the selected color image is higher than brightness of the reference image according to the image histograms, wherein when yes, a color image with an exposure time shorter than an exposure time of the selected color image is selected from the captured color images, or the at least one color sensor is controlled to capture a color image by adopting an exposure time shorter than the exposure time of the selected color image to be used as a combination for comparison with the reference image; and when no, a color image with an exposure time longer than the exposure time of the selected color image is selected from the captured color images, or the at least one color sensor is controlled to capture a color image by adopting an exposure time longer than the exposure time of the selected color image to be used as the combination for comparison with the reference image. ### Claim 17 (depends on 11) The depth map calculation method according to claim 11, further comprising: detecting at least one occlusion lacking color details or texture details in the selected color image; and executing a hole filling algorithm to determine a depth value of the at least one occlusion according to texture details around each of the at least one occlusion provided by the selected IR image and depth values of a plurality of pixels around each of the at least one occlusion. ### Claim 18 (depends on 17) The depth map calculation method according to claim 17, wherein the dual sensor imaging system further comprises an IR projector, and after the step of detecting the at least one occlusion lacking the color details or the texture details in the selected color image, the depth map calculation method further comprises: controlling the IR projector to project an infrared ray to the imaging scene, and controlling the at least one IR sensor to capture an IR image of the imaging scene when the at least one occlusion is detected; and determining the depth value of the at least one occlusion according to the texture details around each of the at least one occlusion provided by the captured IR image and the depth values of the pixels around each of the at least one occlusion. ### Claim 19 (depends on 11) The depth map calculation method according to claim 11, wherein the step of calculating the depth map of the imaging scene by using the selected color image and IR image comprises: capturing a plurality of feature points with strong features in the selected color image and IR image; and calculating the depth map of the imaging scene according to positions of the corresponding feature points in the color image and the IR image. ### Claim 20 (depends on 11) The depth map calculation method according to claim 11, wherein before the step of controlling the at least one color sensor and the at least one IR sensor to respectively capture the color images and the IR images by adopting the exposure conditions suitable for the imaging scene, the depth map calculation method further comprises: controlling at least one of the least one color sensor and the at least one IR sensor to capture at least one standard image of the imaging scene by adopting a standard exposure condition, and using the at least one standard image to identify the imaging scene.