# US 10,321,113 B2 · Processing apparatus, processing system, image pickup apparatus, processing method, and non-transitory computer-readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,321,113 B2 - Document ID: US10321113B2 - Status: Granted - Kind: B2 - Application number: 15/361,684 - Filed: 2016-11-28 - Published: 2017-06-01 - Granted: 2019-06-11 - Earliest claimed: 2015-11-30 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Yuichi Kusumi; Chiaki Inoue; Yoshiaki Ida - Web page: https://patentgrep.com/patent/US10321113B2 ## CPC - H04N 13/133: Equalising the characteristics of different image components, e.g. their average brightness or colour balance - G06T 7/586: from multiple light sources, e.g. photometric stereo - H04N 13/15: for colour aspects of image signals - H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00) - H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects - H04N 13/257: Colour aspects - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G06T 2207/10024: Color image - G06T 2207/10152: Varying illumination - H04N 2013/0077: {Colour aspects} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A processing apparatus includes a luminance information obtainer that obtains luminance information for each color in each of a plurality of images, each of which is imaged respectively when a light source arranged at mutually different positions sequentially irradiates an object with light, and a color setter that sets a color to use in obtaining surface normal information for each region of an image on the basis of the luminance information. ## Claims ### Claim 1 A processing apparatus comprising: a luminance information obtainer that obtains luminance information for each color in each of a plurality of images of an object, each of the plurality of images of the object being respectively imaged when the object is irradiated with light from mutually different positions sequentially; and a color setter that sets a color to use in obtaining surface normal information for each region of an image of the object on the basis of the luminance information, wherein the color setter sets a color from among a plurality of colors to use in obtaining surface normal information on the basis of a value of the luminance information such that a first color among the plurality of colors is set to be used in obtaining surface normal information in a case where the value of the luminance information has a first value, and a second color among the plurality of colors different from the first color is set to be used in obtaining surface normal information in a case where the value of the luminance information has a second value different from the first value. ### Claim 2 (depends on 1) The processing apparatus according to claim 1, further comprising a surface normal obtainer that obtains the surface normal information on the basis of luminance information of the color to use in obtaining surface normal information. ### Claim 3 A processing apparatus comprising: a luminance information obtainer that obtains luminance information for each color in each of a plurality of images of an object, each of the plurality of images of the object being respectively imaged when the object is irradiated with light from mutually different positions sequentially; a color setter that sets a color to use in obtaining surface normal information for each region of an image of the object on the basis of the luminance information; and a surface normal obtainer that obtains the surface normal information on the basis of luminance information of the color to use in obtaining surface normal information, wherein the color setter sets a weight for a color on the basis of a value of the luminance information, and wherein the surface normal obtainer obtains the surface normal information on the basis of a linear sum of luminance information for a color calculated using the weight. ### Claim 4 (depends on 1) The processing apparatus according to claim 1, wherein the color setter sets a color corresponding to luminance information having a maximum value among a plurality of pieces of the luminance information in the region as the color to use in obtaining surface normal information. ### Claim 5 (depends on 1) The processing apparatus according to claim 1, wherein the color setter sets a color corresponding to luminance information having a value larger than a predetermined value in the region as the color to use in obtaining surface normal information. ### Claim 6 (depends on 5) The processing apparatus according to claim 5, wherein the predetermined value is determined on the basis of noise characteristics and a number of the mutually different positions. ### Claim 7 A processing system comprising: a luminance information obtainer that obtains luminance information for each color in each of a plurality of images of an object, each of the plurality of images of the object being respectively imaged when the object is irradiated with light from mutually different positions sequentially; a color setter that sets a color to use in obtaining surface normal information for each region of an image of the object on the basis of the luminance information; and a surface normal obtainer that obtains the surface normal information on the basis of luminance information of the color to use in obtaining surface normal information, wherein the color setter sets a color from among a plurality of colors to use in obtaining surface normal information on the basis of a value of the luminance information such that a first color among the plurality of colors is set to be used in obtaining surface normal information in a case where the value of the luminance information has a first value, and a second color among the plurality of colors different from the first color is set to be used in obtaining surface normal information in a case where the value of the luminance information has a second value different from the first value. ### Claim 8 (depends on 7) The processing system according to claim 7, further comprising a light source unit that includes three or more light sources each different in a position. ### Claim 9 An image pickup apparatus comprising: an image pickup unit that images an object; a luminance information obtainer that obtains luminance information for each color in each of a plurality of images of an object, each of the plurality of images of the object being imaged by the image pickup unit respectively when the object is irradiated with light from mutually different positions sequentially; a color setter that sets a color to use in obtaining surface normal information for each region of an image of the object on the basis of the luminance information; and a surface normal obtainer that obtains the surface normal information on the basis of luminance information of the color to use in obtaining surface normal information, wherein the color setter sets a color from among a plurality of colors to use in obtaining surface normal information on the basis of a value of the luminance information such that a first color among the plurality of colors is set to be used in obtaining surface normal information in a case where the value of the luminance information has a first value, and a second color among the plurality of colors different from the first color is set to be used in obtaining surface normal information in a case where the value of the luminance information has a second value different from the first value. ### Claim 10 (depends on 9) The image pickup apparatus according to claim 9, further comprising a light source unit that includes three or more light sources each different in a position. ### Claim 11 (depends on 9) The image pickup apparatus according to claim 9, wherein the image pickup unit includes an image pickup element adopting a Bayer array. ### Claim 12 (depends on 11) The image pickup apparatus according to claim 11, wherein the surface normal obtainer obtains surface normal information of a first pixel corresponding to a color different from the color to use in obtaining surface normal information on the basis of surface normal information of one or more second pixels near the first pixel. ### Claim 13 A processing method comprising: obtaining luminance information for each color in each of a plurality of images of an object, each of the plurality of images of the object being respectively imaged when the object is irradiated with light from mutually different positions sequentially; and setting a color to use in obtaining surface normal information for each region of an image of the object on the basis of the luminance information, wherein the setting the color sets a color from among a plurality of colors to use in obtaining surface normal information on the basis of a value of the luminance information such that a first color among the plurality of colors is set to be used in obtaining surface normal information in a case where the value of the luminance information has a first value, and a second color among the plurality of colors different from the first color is set to be used in obtaining surface normal information in a case where the value of the luminance information has a second value different from the first value. ### Claim 14 (depends on 13) The processing method according to claim 13, further comprising: calculating a plurality of pieces of surface normal information on the basis of the luminance information; and selecting surface normal information from among the plurality of pieces of surface normal information on the basis of the color to use in obtaining surface normal information. ### Claim 15 A non-transitory computer-readable storage medium configured to store a computer program that enables a computer to execute a processing method, wherein the processing method includes: obtaining luminance information for each color in each of a plurality of images of an object, each of the plurality of images of the object being respectively imaged when the object is irradiated with light from mutually different positions sequentially; and setting a color to use in obtaining surface normal information for each region of an image of the object on the basis of the luminance information, wherein the setting the color sets a color from among a plurality of colors to use in obtaining surface normal information on the basis of a value of the luminance information such that a first color among the plurality of colors is set to be used in obtaining surface normal information in a case where the value of the luminance information has a first value, and a second color among the plurality of colors different from the first color is set to be used in obtaining surface normal information in a case where the value of the luminance information has a second value different from the first value.