# US 12,112,419 B2 · Image processing apparatus, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,419 B2 - Document ID: US12112419B2 - Status: Granted - Kind: B2 - Application number: 17/701,107 - Filed: 2022-03-22 - Published: 2022-07-07 - Granted: 2024-10-08 - Earliest claimed: 2019-06-03 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Daichi Adachi - Web page: https://patentgrep.com/patent/US12112419B2 ## CPC - G06T 15/00: Three-dimensional [3D] image rendering - G06T 15/20: Perspective computation - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 15/205: {Image-based rendering} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The technique of this disclosure suppresses a reduction in visibility of a predetermined object in virtual viewpoint image data. An image processing apparatus includes: an image capturing information acquisition unit configured to acquire image capturing information indicating a position and orientation of each of a plurality of image capturing apparatuses; an object information acquisition unit configured to acquire object information indicating a position and orientation of an object to be captured by the image capturing apparatuses, the object having a specific viewing angle; and a determination unit configured to determine, based on the acquired image capturing information and the position and orientation of the object indicated by the acquired object information, an image to be used for generating a virtual viewpoint image according to a position and orientation of a virtual viewpoint among a plurality of images based on capturing by the image capturing apparatuses. ## Claims ### Claim 1 An image processing system comprising: one or more memories storing instructions; and one or more processors that execute the instructions to: obtain images captured by a plurality of image capturing apparatuses; obtain position information indicating position of a display surface of a display apparatus captured by one or more image capturing apparatuses whose view directions have a predetermined relationship with a normal direction of the display surface of the display apparatus among the plurality of image capturing apparatuses; and generate a virtual viewpoint image showing the display surface of the display apparatus based on the obtained images and the position information. ### Claim 2 (depends on 1) The image processing system according to claim 1, wherein a difference between an optical axis direction according to an orientation of an image capturing apparatus, among the one or more image capturing apparatuses, capturing the image showing the display surface of the display apparatus and the normal direction to the display surface of the display apparatus is less than a difference between a view direction from the virtual viewpoint and the normal direction to the display surface of the display apparatus. ### Claim 3 (depends on 1) The image processing system according to claim 1, wherein: the display surface is a light-emitting surface, and the normal direction to the display surface of the display apparatus is a direction perpendicular to the light-emitting surface. ### Claim 4 (depends on 1) The image processing system according to claim 1, wherein an image capturing apparatus, among the one or more capturing apparatuses, capturing the image showing the display surface of the display apparatus has a smallest difference between the normal direction to the display surface of the display apparatus and an optical axis direction according to an orientation of the image capturing apparatus. ### Claim 5 (depends on 1) The image processing system according to claim 1, wherein: the one or more processors execute the instructions to further set a priority of each image capturing apparatus, among the one or more image capturing apparatuses, based on the obtained image capturing information and the obtained display apparatus information, and an image, among the obtained images used for generating the virtual viewpoint image is further determined based on the set priority. ### Claim 6 (depends on 5) The image processing system according to claim 5, wherein: as a difference between an optical axis direction according to an orientation of an image capturing apparatus, among the one or more image capturing apparatuses, and the normal direction to the display surface of the display apparatus is smaller, a higher priority is set for the image capturing apparatus, and an image captured by an image capturing apparatus, among the plurality of image capturing apparatuses, for which a high priority is set, is preferentially used for generating the virtual viewpoint image. ### Claim 7 (depends on 5) The image processing system according to claim 5, wherein the priority is determined based on the image capturing information, the display apparatus information, and viewpoint information for specifying a position of the virtual viewpoint and a view direction from the virtual viewpoint. ### Claim 8 (depends on 1) The image processing system according to claim 1, wherein: the display apparatus information includes characteristic information for specifying a luminance of the display surface, and an image, among the obtained images, to be used for generating the virtual viewpoint image is further determined from the characteristic information. ### Claim 9 (depends on 1) The image processing system according to claim 1, wherein the image showing the display surface of the display apparatus is used for determining a color of a three-dimensional model of the display apparatus. ### Claim 10 (depends on 1) The image processing system according to claim 1, wherein: the one or more processors execute the instructions to further obtain virtual viewpoint information for specifying a position of the virtual viewpoint and a view direction from the virtual viewpoint, and the virtual viewpoint image is generated based on, the image showing the display surface of the display apparatus, among the obtained images, and the obtained viewpoint information. ### Claim 11 (depends on 1) The image processing system according to claim 1, wherein a color of the display surface of the display apparatus in the virtual viewpoint image is determined based on one or more images showing a display surface of a display apparatus from the images. ### Claim 12 (depends on 1) The image processing system according to claim 1, wherein the one or more processors further execute the instructions to obtain the image showing the display surface of the display apparatus. ### Claim 13 (depends on 1) The image processing system according to claim 1, wherein the predetermined relationship is that view directions of the one or more image capturing apparatuses are parallel or nearly parallel to the normal direction of the display surface of the display apparatus. ### Claim 14 (depends on 1) The image processing system according to claim 1, wherein the virtual viewpoint image is generated based on images captured by the one or more image capturing apparatuses included in the obtained images. ### Claim 15 An image processing method comprising: obtaining images captured by a plurality of image capturing apparatuses; obtaining position information indicating position of a display surface of a display apparatus captured by one or more image capturing apparatuses whose view directions have a predetermined relationship with a normal direction of the display surface of the display apparatus among the plurality of image capturing apparatuses; and generating a virtual viewpoint image showing the display surface of the display apparatus based on the obtained images and the position information. ### Claim 16 (depends on 15) The image processing method according to claim 15, wherein a difference between an optical axis direction according to an orientation of an image capturing apparatus, among the one or more image capturing apparatuses, capturing the image showing the display surface of the display apparatus and the normal direction to the display surface of the display apparatus is less than a difference between a view direction from the virtual viewpoint and the normal direction to the display surface of the display apparatus. ### Claim 17 A non-transitory computer readable storage medium storing a program executable by a computer to execute an image processing method comprising: obtaining images captured by a plurality of image capturing apparatuses; obtaining position information indicating position of a display surface of a display apparatus captured by one or more image capturing apparatuses whose view directions have a predetermined relationship with a normal direction of the display surface of the display apparatus among the plurality of image capturing apparatuses; and generating a virtual viewpoint image showing the display surface of the display apparatus based on the obtained images and the position information.