# US 10,122,980 B2 · Image processing apparatus, image processing method, and non-transitory computer-readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,980 B2 - Document ID: US10122980B2 - Status: Granted - Kind: B2 - Application number: 15/051,873 - Filed: 2016-02-24 - Published: 2016-09-08 - Granted: 2018-11-06 - Earliest claimed: 2015-03-02 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Satoshi Nakashima - Web page: https://patentgrep.com/patent/US10122980B2 ## CPC - H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control - G02B 27/017: {Head mounted} - H04N 13/344: with head-mounted left-right displays - G02B 2027/0112: {comprising device for genereting colour display} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - H04N 2209/046: Colour interpolation to calculate the missing colour values - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract A color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device is acquired, a color temperature that is common between the first image sensing device and the second image sensing device is decided based on the acquired color temperatures, and color information in an image sensed by the first image sensing device and an image sensed by the second image sensing device is adjusted based on the decided color temperature. ## Claims ### Claim 1 An image processing apparatus comprising: a processor; and a memory storing instructions which, when executed by the processor, cause the image processing apparatus to: acquire a color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device; select a way to decide a common color temperature that is common to the first image sensing device and the second image sensing device based on a difference between the color temperatures acquired from the first and the second image sensing devices; decide the common color temperature in accordance with the selected way; and adjust color information of at least one of the first and the second image sensing devices so that the first and the second image sensing devices use the common color temperature. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the adjustment of color information includes adjusting each of R, G, and B signals in an image sensed by the first image sensing device and an image sensed by the second image sensing device based on the common color temperature decided. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the selected way includes a way to decide the common color temperature based on a statistic of the color temperatures acquired. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the selected way includes a way to decide the common color temperature based on a maximum or a minimum of the color temperatures acquired. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the instructions, when executed by the processor, further cause the image processing apparatus to obtain a difference between a maximum color temperature and a minimum color temperature among the color temperatures acquired for each of the first sensed image and the second sensed image, and wherein the selection of the way to decide the common color temperature is based on whether the difference is in an allowable range. ### Claim 6 (depends on 5) The image processing apparatus according to claim 5, wherein the selection of the way to decide the common color temperature differs in a case where the difference is in the allowable range to in a case where the difference is not in the allowable range. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein if the difference is in the allowable range, the common color temperature is decided to be an average color temperature of all or a portion of the color temperatures for each of the first sensed image and the second sensed image, and if the difference is not in the allowable range, the common color temperature is decided to be a color temperature that is closest to a color temperature set in advance as a standard color temperature among all or a portion of the color temperatures for each of the first sensed image and the second sensed image. ### Claim 8 (depends on 6) The image processing apparatus according to claim 6, wherein if the difference is in the allowable range, the common color temperature is decided to be an average color temperature of all or a portion of the color temperatures acquired for each of the first sensed image and the second sensed image, and if the difference is not in the allowable range, the common color temperature is decided to be one color temperature obtained from the color temperatures acquired for each of the first sensed image and the second sensed image. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the one color temperature obtained from the color temperatures acquired for each of the first sensed image and the second sensed image is a color temperature determined based on a statistic of the color temperatures acquired for each of the first sensed image and the second sensed image. ### Claim 10 (depends on 7) The image processing apparatus according to claim 7, wherein an average color temperature of a portion of the color temperatures acquired for each of the first sensed image and the second sensed image is an average color temperature of color temperatures remaining after removing, from the color temperatures acquired for each of the first sensed image and the second sensed image, a color temperature for which a difference from an average color temperature of the color temperatures is largest. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the acquisition of the color temperature of each of the first sensed image and the second sensed image includes: obtaining an average pixel value for each color component of the respective sensed image, and acquiring the color temperature of the respective sensed image using the average pixel values. ### Claim 12 (depends on 11) The image processing apparatus according to claim 11, wherein the acquisition of the color temperature of each of the first sensed image and the second sensed image includes: obtaining the average pixel value of an R component and an average pixel value of a B component of the respective sensed image, and acquiring a color temperature that corresponds to (the average pixel value of the B component) / (the average pixel value of the R component). ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing apparatus is a head-mounted display equipped with the first image sensing device, the second image sensing device, and a display apparatus. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the instructions, when executed by the processor, further cause the image processing apparatus to: derive a position and orientation of the first image sensing device, and compose a virtual object, which is generated based on the derived position and orientation of the first image sensing device, with the first sensed image. ### Claim 15 (depends on 14) The image processing apparatus according to claim 14, wherein the derivation of the position and orientation of the first image sensing device is based on the second sensed image. ### Claim 16 (depends on 1) The image processing apparatus according to claim 1, wherein a type of a sensor of the first image sensing device differs from a type of a sensor of the second image sensing device. ### Claim 17 (depends on 16) The image processing apparatus according to claim 16, wherein the sensor of the first image sensing device is of a rolling shutter method, and the sensor of the second image sensing device is of a global shutter method. ### Claim 18 An image processing method comprising: acquiring a color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device; selecting a way for deciding a common color temperature that is common to the first image sensing device and the second image sensing device based on a difference between the color temperatures acquired from the first and the second image sensing devices; deciding the common color temperature in accordance with the selected way; and adjusting color information of at least one of the first and the second image sensing devices so that the first and the second image sensing devices use the common color temperature. ### Claim 19 A non-transitory computer-readable storage medium storing a computer program for causing a computer to: acquire a color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device; select a way to decide a common color temperature that is common to the first image sensing device and the second image sensing device based on a difference between the color temperatures acquired from the first and the second image sensing devices; decide the common color temperature in accordance with the selected way; and adjust color information of at least one of the first and the second image sensing devices so that the first and the second image sensing devices use the common color temperature. ### Claim 20 (depends on 3) The apparatus according to claim 3, wherein the decision of the common color temperature is based on the statistic of the color temperatures acquired, if a difference between the color temperatures acquired is less than a predetermined value. ### Claim 21 (depends on 4) The apparatus according to claim 4, wherein the decision of the common color temperature is based on the maximum or the minimum of the color temperatures acquired, if a difference between the color temperatures acquired is not less than a predetermined value. ### Claim 22 (depends on 4) The apparatus according to claim 4, wherein the decision of the common color temperature is based on a color temperature closer to a predetermined reference value among the maximum or the minimum of the color temperatures acquired by the acquisition unit. ### Claim 23 An image processing apparatus comprising: a processor; and a memory storing instructions which, when executed by the processor, cause the image processing apparatus to: acquire a color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device; decide, in accordance with a difference between the acquired color temperature of the first sensed image and the acquired color temperature of the second sensed image, a common color temperature that is common to the first image sensing device and the second image sensing device based on the color temperatures acquired; and adjust color information of at least one of the first and the second image sensing devices so that the first and the second image sensing devices use the common color temperature. ### Claim 24 (depends on 23) The image processing apparatus according to claim 23, wherein if the difference is in a predetermined range, the common color temperature is decided to be an average color temperature of the color temperatures of the first sensed image and the second sensed image. ### Claim 25 (depends on 23) The image processing apparatus according to claim 23, wherein if the difference is not in a predetermined range, the common color temperature is decided to be a color temperature that is closest to a predetermined color temperature among the color temperatures of the first sensed image and the second sensed image. ### Claim 26 An image processing method comprising: acquiring a color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device; deciding, in accordance with a difference between the acquired color temperature of the first sensed image and the acquired color temperature of the second sensed image, a common color temperature that is common to the first image sensing device and the second image sensing device based on the color temperatures acquired; and adjusting color information of at least one of the first and the second image sensing devices so that the first and the second image sensing devices use the common color temperature. ### Claim 27 A non-transitory computer-readable storage medium storing a computer program for causing a computer to: acquire a color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device; decide, in accordance with a difference between the acquired color temperature of the first sensed image and the acquired color temperature of the second sensed image, a common color temperature that is common to the first image sensing device and the second image sensing device based on the color temperatures acquired; and adjust color information of at least one of the first and the second image sensing devices so that the first and the second image sensing devices use the common color temperature.