# US 2022/0036591 A1 · Image processing apparatus, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036591 A1 - Document ID: US20220036591A1 - Status: Published application - Kind: A1 - Application number: 17/369,996 - Filed: 2021-07-08 - Published: 2022-02-03 - Earliest claimed: 2020-07-31 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Hiroyasu Ito - Web page: https://patentgrep.com/patent/US20220036591A1 ## CPC - G06T 7/90: Determination of colour characteristics - G06T 15/205: {Image-based rendering} - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/15: for colour aspects of image signals - H04N 13/257: Colour aspects - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - G06T 2207/20224: Image subtraction - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract With an image processing apparatus, a captured image to be used for obtaining color information of an object is specified based on a designated virtual viewpoint from among multiple captured images obtained by capturing images of the object from different positions. Color information of the object in a virtual viewpoint image is determined based on the color information of the object obtained from the specified captured image. The virtual viewpoint image is generated based on the determined color information of the object. In a case where a structure occluding the object is included in the specified captured image, a captured image that is different from the specified captured image is specified as the captured image to be used for obtaining the color information of the object, based on color information of the structure. ## Claims ### Claim 1 An image processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to: specify a captured image to be used for obtaining color information of an object, based on a designated virtual viewpoint, from among a plurality of captured images obtained by a plurality of image capturing units that capture the object from different positions; determine color information of the object in a virtual viewpoint image, based on the color information of the object obtained from the captured image specified in the specifying; and generate the virtual viewpoint image, based on the color information of the object determined in the determining, wherein, in the specifying, in a case where a structure occluding the object is included in the specified captured image, a captured image that is different from the specified captured image is specified as the captured image to be used for obtaining the color information of the object, based on color information of the structure. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein, in the specifying, a captured image corresponding to an image capturing unit having a field of view that is the closest to a field of view of the virtual viewpoint is specified as the captured image to be used for obtaining the color information of the object. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein, in the specifying, in a case where the color information of the object obtained from the specified captured image is included in prohibited color information, which is set based on the color information of the structure and is set for prohibiting coloring on the object, another captured image that is different from the specified captured image is selected in an order from a captured image of an image capturing unit having a field of view that is closer to the field of view of the virtual viewpoint, and, in a case where color information of the object obtained from the selected other captured image is not included in the prohibited color information, the selected other captured image is specified as the captured image to be used for obtaining the color information of the object. ### Claim 4 (depends on 3) The image processing apparatus according to claim 3, wherein, in the specifying, in a case where all of the color information of the object obtained from the specified captured images are included in the prohibited color information, the captured image of the image capturing unit having the field of view that is the closest to the field of view of the virtual viewpoint is specified as the captured image to be used for obtaining the color information of the object. ### Claim 5 (depends on 3) The image processing apparatus according to claim 3, wherein, in the specifying, in a case where all of the color information of the object obtained from the specified captured images are included in the prohibited color information, a captured image of the object in a state where the structure is not included in an image capturing area of the plurality of image capturing units is specified as the captured image to be used for obtaining the color information of the object. ### Claim 6 (depends on 3) The image processing apparatus according to claim 3, wherein, of a target pixel and peripheral pixels thereof in a unit area, the prohibited color information is a pattern that is set for a plurality of pixels at least including the target pixel. ### Claim 7 (depends on 2) The image processing apparatus according to claim 2, wherein, in the specifying, in a case where a position of obtaining the color information of the object in the specified captured image is included in a prohibited area, which is set based on an area having color information whose color difference from the object is relatively large within an area representing the structure and is set for prohibiting coloring on the object, another captured image that is different from the specified captured image is selected in an order from a captured image of an image capturing unit having a field of view that is closer to the field of view of the virtual viewpoint, and, in a case where a position of obtaining the color information of the object in the selected other captured image is not included in the prohibited area, the selected other captured image is specified as the captured image to be used for obtaining the color information of the object. ### Claim 8 (depends on 7) The image processing apparatus according to claim 7, wherein, in the specifying, in a case where all positions of obtaining the color information of the object in the specified captured images are included in the prohibited area, the captured image of the image capturing unit having the field of view that is the closest to the field of view of the virtual viewpoint is specified as the captured image to be used for obtaining the color information of the object. ### Claim 9 (depends on 7) The image processing apparatus according to claim 7, wherein, in the specifying, in a case where all positions of obtaining the color information of the object in the specified captured images are included in the prohibited area, a captured image of the object in a state where the structure is not included in an image capturing area of the plurality of image capturing units is specified as the captured image to be used for obtaining the color information of the object. ### Claim 10 (depends on 2) The image processing apparatus according to claim 2, wherein, in the specifying, in a case where the color information of the object obtained from the specified captured image is included in prohibited color information, which is set based on the color information of the structure and is set for prohibiting coloring on the object, or in a case where a position of obtaining the color information of the object in the specified captured image is included in a prohibited area, which is set based on an area having color information whose color difference from the object is relatively large within an area representing the structure and is set for prohibiting coloring on the object, another captured image that is different from the specified captured image is selected in an order from a captured image of an image capturing unit having a field of view that is closer to the field of view of the virtual viewpoint, and, in a case where the color information of the object obtained from the selected other captured image is not included in the prohibited color information and a position of obtaining the color information of the object in the selected other captured image is not included in the prohibited area, the selected other captured image is specified as the captured image to be used for obtaining the color information of the object. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein, in the specifying, in a case where all of the color information of the object obtained from the specified captured images are included in the prohibited color information or in a case where all of the positions of obtaining the color information of the object in the specified captured images are included in the prohibited area, the captured image of the image capturing unit having the field of view that is the closest to the field of view of the virtual viewpoint is specified as the captured image to be used for obtaining the color information of the object. ### Claim 12 (depends on 10) The image processing apparatus according to claim 10, wherein, in the specifying, in a case where all of the color information of the object obtained from the specified captured images are included in the prohibited color information or in a case where all of the positions of obtaining the color information of the object in the specified captured images are included in the prohibited area, a captured image of the object in a state where the structure is not included in an image capturing area of the plurality of image capturing units is specified as the captured image to be used for obtaining the color information of the object. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the structure is in a shape of a net. ### Claim 14 An image processing method comprising: specifying a captured image to be used for obtaining color information of an object, based on a designated virtual viewpoint, from among a plurality of captured images obtained by a plurality of image capturing units that capture the object from different positions; determining color information of the object in a virtual viewpoint image, based on the color information of the object obtained from the specified captured image; and generating the virtual viewpoint image, based on the determined color information of the object, wherein, in the specifying, in a case where a structure occluding the object is included in the specified captured image, a captured image that is different from the specified captured image is specified as the captured image to be used for obtaining the color information of the object, based on color information of the structure. ### Claim 15 A non-transitory computer readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising: specifying a captured image to be used for obtaining color information of an object, based on a designated virtual viewpoint, from among a plurality of captured images obtained by a plurality of image capturing units that capture the object from different positions; determining color information of the object in a virtual viewpoint image, based on the color information of the object obtained from the specified captured image; and generating the virtual viewpoint image, based on the determined color information of the object, wherein, in the specifying, in a case where a structure occluding the object is included in the specified captured image, a captured image that is different from the specified captured image is specified as the captured image to be used for obtaining the color information of the object, based on color information of the structure.