# US 12,192,649 B2 · Infrared-based processing of an image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,649 B2 - Document ID: US12192649B2 - Status: Granted - Kind: B2 - Application number: 17/334,182 - Filed: 2021-05-28 - Published: 2022-12-01 - Granted: 2025-01-07 - Earliest claimed: 2021-05-28 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Wen-Chun Feng; Wei-Chih Liu; Yi-Chun Hsu; Tai-Hsiang Jen - Web page: https://patentgrep.com/patent/US12192649B2 ## CPC - H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control - G01J 5/60: using determination of colour temperature - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - H04N 23/20: for generating image signals from infrared radiation only - H04N 23/71: Circuitry for evaluating the brightness variation - G01J 2005/0077: {Imaging} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract A multi-point measurement of a scene captured in an image frame may be used to process the image frame, such as by applying white balance corrections to the image frame. In some examples, the image frame may be segmented into portions that are illuminated by different illumination sources. Different portions of the image frame may be white balanced differently based on the color temperature of the illumination source for the corresponding portion. Infrared measurements of multiple points in the scene may be used to determine a characteristic of the illumination source of different portions of the scene. For example, a picture that includes indoor and outdoor portions may be illuminated by at least two illumination sources that produce different infrared measurements values. White balancing may be applied differently to these two portions to correct for color temperature of the different sources. ## Claims ### Claim 1 A method, comprising: receiving a first image frame and a corresponding set of infrared measurement values; dividing the first image frame into a plurality of portions based on the values included in the set of infrared measurement values, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, the first portion having infrared measurement values of the corresponding set of infrared measurement values that are below a first threshold, and the second portion having infrared measurement values of the corresponding set of infrared measurement values that are above the first threshold; and based on the division, processing the first image frame as a unit by applying a first white balancing to the first portion and applying a second white balancing to the second portion. ### Claim 2 (depends on 1) The method of claim 1, wherein: the dividing comprises: determining an indoor region as the first portion of the first image frame; and determining an outdoor region as the second portion of the first image frame; the applying of the first white balancing to the first portion comprises applying a lower correlated color temperature (CCT) to the indoor region; and the applying of the second white balancing to the second portion comprises applying a higher correlated color temperature (CCT) to the outdoor region. ### Claim 3 (depends on 1) The method of claim 1, wherein: pixels in the first portion have color values within a first threshold distance; and pixels in the second portion have color values within a second threshold distance. ### Claim 4 (depends on 1) The method of claim 1, wherein: applying the first white balancing comprises applying the first white balancing based on image content of the first portion; and applying the second white balancing comprises applying the second white balancing based on image content of the first portion. ### Claim 5 (depends on 1) The method of claim 1, wherein the dividing comprises: performing a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; identifying a first set of pixels of the first image frame as the first portion of the first image frame based on identifying a first contiguous set of the corresponding set of infrared measurement values that are below the first threshold and a first set of the plurality of features corresponding to the first contiguous set; and identifying a second set of pixels of the first image frame as the second portion of the first image frame based on identifying a second contiguous set of the corresponding set of infrared measurement values that are above the first threshold and a second set of the plurality of features corresponding to the second contiguous set. ### Claim 6 (depends on 1) The method of claim 1, wherein the processing comprises: determining white balancing weights for the first image frame based on the set of infrared measurement values; and applying the respective white balancing based on the white balancing weights. ### Claim 7 (depends on 1) The method of claim 1, wherein receiving the corresponding set of infrared measurement values comprises: receiving a set of light detection and ranging (LiDAR) measurements from a LiDAR sensor; or receiving a set of time of flight (ToF) measurements from a time-of-flight sensor. ### Claim 8 A device, comprising: a processor; and a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform operations comprising: receiving a first image frame and a corresponding set of infrared measurement values; dividing the first image frame into a plurality of portions based on the values included in the set of infrared measurement values, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, the first portion having infrared measurement values of the corresponding set of infrared measurement values that are below a first threshold, and the second portion having infrared measurement values of the corresponding set of infrared measurement values that are above the first threshold; and based on the division, processing the first image frame by applying a first white balancing to the first portion and applying a second white balancing to the second portion. ### Claim 9 (depends on 8) The device of claim 8, wherein: the dividing comprises: determining an indoor region as the first portion of the first image frame; and determining an outdoor region as the second portion of the first image frame; the applying of the first white balancing to the first portion comprises applying a lower correlated color temperature (CCT) to the indoor region; and the applying of the second white balancing to the second portion comprises applying a higher correlated color temperature (CCT) to the outdoor region. ### Claim 10 (depends on 8) The device of claim 8, wherein: pixels in the first portion have color values within a first threshold distance; and pixels in the second portion have color values within a second threshold distance. ### Claim 11 (depends on 8) The device of claim 8, wherein: applying the first white balancing comprises applying the first white balancing based on image content of the first portion; and applying the second white balancing comprises applying the second white balancing based on image content of the first portion. ### Claim 12 (depends on 8) The device of claim 8, wherein dividing comprises: performing a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; identifying a first set of pixels of the first image frame as the first portion of the first image frame based on identifying a first contiguous set of the corresponding set of infrared measurement values that are below the first threshold and a first set of the plurality of features corresponding to the first contiguous set; and identifying a second set of pixels of the first image frame as the second portion of the first image frame based on identifying a second contiguous set of the corresponding set of infrared measurement values that are above the first threshold and a second set of the plurality of features corresponding to the second contiguous set. ### Claim 13 (depends on 8) The device of claim 8, wherein the processing the first image frame comprises: determining white balancing weights for the first image frame based on the set of infrared measurement values; and applying the respective white balancing based on the white balancing weights. ### Claim 14 (depends on 8) The device of claim 8, wherein receiving the corresponding set of infrared measurement values comprises: receiving a set of light detection and ranging (LiDAR) measurements from a LiDAR sensor; or receiving a set of time of flight (ToF) measurements from a time-of-flight sensor. ### Claim 15 (depends on 8) The device of claim 8, wherein at least two of the plurality of portions have a different number of pixels as each other. ### Claim 16 A non-transitory computer-readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations comprising: receiving a first image frame and a corresponding set of infrared measurement values; dividing the first image frame into a plurality of portions based on the values included in the set of infrared measurement values, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, the first portion having infrared measurement values of the corresponding set of infrared measurement values that are below a first threshold, and the second portion having infrared measurement values of the corresponding set of infrared measurement values that are above the first threshold; and based on the division, processing the first image frame as a unit by applying a first white balancing to the first portion and applying a second white balancing to the second portion. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein: the dividing comprises: determining an indoor region as the first portion of the first image frame; and determining an outdoor region as the second portion of the first image frame; the applying of the first white balancing to the first portion comprises applying a lower correlated color temperature (CCT) to the indoor region; and the applying of the second white balancing to the second portion comprises applying a higher correlated color temperature (CCT) to the outdoor region. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein: pixels in the first portion have color values within a first threshold distance; and pixels in the second portion have color values within a second threshold distance. ### Claim 19 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein: applying the first white balancing comprises applying the first white balancing based on image content of the first portion; and applying the second white balancing comprises applying the second white balancing based on image content of the first portion. ### Claim 20 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein dividing comprises: performing a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; identifying a first set of pixels of the first image frame as the first portion of the first image frame based on identifying a first contiguous set of the corresponding set of infrared measurement values that are below the first threshold and a first set of the plurality of features corresponding to the first contiguous set; and identifying a second set of pixels of the first image frame as the second portion of the first image frame based on identifying a second contiguous set of the corresponding set of infrared measurement values that are above the first threshold and a second set of the plurality of features corresponding to the second contiguous set. ### Claim 21 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the processing the first image frame comprises: determining white balancing weights for the first image frame based on the set of infrared measurement values; and applying the respective white balancing based on the white balancing weights. ### Claim 22 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein receiving the corresponding set of infrared measurement values comprises: receiving a set of light detection and ranging (LiDAR) measurements from a LiDAR sensor; or receiving a set of time of flight (ToF) measurements from a time-of-flight sensor. ### Claim 23 A device, comprising: a first image sensor; a first multi-point infrared (IR) detector; a processor coupled to the first image sensor and to the first multi-point IR detector; and a memory coupled to the processor, wherein the processor is configured to perform steps comprising: receiving a first image frame from the first image sensor and a corresponding set of infrared measurement values from the first multi-point IR detector; dividing the first image frame into a plurality of portions based on the values included in the set of infrared measurement values, the plurality of portions including a first portion illuminated by a first light source and a second portion illuminated by a second light source, the first portion having infrared measurement values of the corresponding set of infrared measurement values that are below a first threshold, and the second portion having infrared measurement values of the corresponding set of infrared measurement values that are above the first threshold; and based on the division, processing the first image frame as a unit by applying a first white balancing to the first portion and applying a second white balancing to the second portion. ### Claim 24 (depends on 23) The device of claim 23, wherein: the dividing comprises: determining an indoor region as the first portion of the first image frame; and determining an outdoor region as the second portion of the first image frame; the applying of the first white balancing to the first portion comprises applying a lower correlated color temperature (CCT) to the indoor region; and the applying of the second white balancing to the second portion comprises applying a higher correlated color temperature (CCT) to the outdoor region. ### Claim 25 (depends on 23) The device of claim 23, wherein: pixels in the first portion have color values within a first threshold distance; and pixels in the second portion have color values within a second threshold distance. ### Claim 26 (depends on 23) The device of claim 23, wherein: applying the first white balancing comprises applying the first white balancing based on image content of the first portion; and applying the second white balancing comprises applying the second white balancing based on image content of the first portion. ### Claim 27 (depends on 23) The device of claim 23, wherein dividing comprises: performing a computer vision analysis of the first image frame to identify a plurality of features in the first image frame; identifying a first set of pixels of the first image frame as the first portion of the first image frame based on identifying a first contiguous set of the corresponding set of infrared measurement values that are below the first threshold and a first set of the plurality of features corresponding to the first contiguous set; and identifying a second set of pixels of the first image frame as the second portion of the first image frame based on identifying a second contiguous set of the corresponding set of infrared measurement values that are above the first threshold and a second set of the plurality of features corresponding to the second contiguous set. ### Claim 28 (depends on 23) The device of claim 23, wherein the multi-point IR detector comprises a time of flight (ToF) measurement device, wherein receiving the corresponding set of infrared measurement values comprises receiving a set of ToF measurements from the time-of-flight measurement device. ### Claim 29 (depends on 23) The device of claim 23, wherein the multi-point IR detector comprises a light detection and ranging (LiDAR) device, wherein receiving the corresponding set of infrared measurement values comprises receiving a set of light detection and ranging (LiDAR) measurements from the LiDAR device.