# US 2019/0246089 A1 · Information processing apparatus and method of controlling the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0246089 A1 - Document ID: US20190246089A1 - Status: Published application - Kind: A1 - Application number: 16/262,251 - Filed: 2019-01-30 - Published: 2019-08-08 - Earliest claimed: 2018-02-05 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Mitsuru Maeda - Web page: https://patentgrep.com/patent/US20190246089A1 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - G06T 7/55: from multiple images - G06T 15/205: {Image-based rendering} - H04N 13/189: Recording image signals; Reproducing recorded image signals - H04N 23/60: Control of cameras or camera modules - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An information processing apparatus that performs processing to generate a virtual viewpoint image based on a plurality of captured images obtained by a plurality of cameras, comprises an obtaining unit that obtains visibility information which is related to components of a three-dimensional shape model of an object, wherein the visibility information is configured to be used for identifying whether pixels corresponding to the components are included in a captured image for each of not less than two captured images of the plurality of captured images, a selection unit that selects, based on the visibility information, from material data which is based on the captured image obtained by a camera included in the plurality of cameras and is stored in a predetermined storage unit, data to be used to generate the virtual viewpoint image. ## Claims ### Claim 1 An information processing apparatus that performs processing to generate a virtual viewpoint image based on a plurality of captured images obtained by performing image capturing from a plurality of directions by a plurality of cameras, comprising: an obtaining unit configured to obtain visibility information which is related to components of a three-dimensional shape model of an object to be captured by at least one of the plurality of cameras, wherein the visibility information is configured to be used for identifying whether pixels corresponding to the components are included in a captured image for each of not less than two captured images of the plurality of captured images; and a selection unit configured to select, based on the visibility information obtained by the obtaining unit, from material data which is based on the captured image obtained by a camera included in the plurality of cameras and is stored in a predetermined storage unit, data to be used to generate the virtual viewpoint image. ### Claim 2 (depends on 1) The apparatus according to claim 1, wherein the material data stored in the predetermined storage unit includes at least one of partial image data extracted from the captured image and component data of the three-dimensional shape model which is generated based on the plurality of the captured images. ### Claim 3 (depends on 1) The apparatus according to claim 1, further comprising: a determination unit configured to determine, based on the visibility information obtained by the obtaining unit, a priority of the material data stored in the predetermined storage unit, wherein the selection unit selects, based on the priority determined by the determination unit, the data to be used for the generation of the virtual viewpoint image. ### Claim 4 (depends on 3) The apparatus according to claim 3, wherein the determination unit determines, based on the number of the components of the three-dimensional shape model corresponding to the pixels of the image data, a priority of image data stored as the material data in the predetermined storage unit. ### Claim 5 (depends on 4) The apparatus according to claim 4, wherein the larger the number of the components of the three-dimensional shape model corresponding to the pixels of the image data stored as the material data in the predetermined storage unit, the higher a priority that the determination unit determines to set to the image data. ### Claim 6 (depends on 3) The apparatus according to claim 3, wherein the determination unit determines, based on the number of the captured images including the pixels corresponding to the components from the plurality of captured images obtained by the plurality of cameras, a priority of the component data of the three-dimensional shape model stored as the material data in the predetermined storage unit. ### Claim 7 (depends on 6) The apparatus according to claim 6, wherein the smaller the number of the captured images including the pixels corresponding to the components of the three-dimensional shape model stored as the material data in the predetermined storage unit among the plurality of captured images obtained by the plurality of the cameras, the higher a priority that the determination unit determines to set to the component data. ### Claim 8 (depends on 1) The apparatus according to claim 1, wherein the selection unit selects the data to be used for the generation of the virtual viewpoint image so a total amount of the data to be used for the generation of the virtual viewpoint image will not exceed a predetermined data capacity. ### Claim 9 (depends on 8) The apparatus according to claim 8, wherein the predetermined data capacity is a data capacity related to a storage unit or a line included in a system for generating the virtual viewpoint image. ### Claim 10 (depends on 1) The apparatus according to claim 1, further comprising: an output unit configured to output, to a generation apparatus configured to generate the virtual viewpoint image, the data selected by the selection unit. ### Claim 11 (depends on 1) The apparatus according to claim 1, further comprising: a generating unit configured to generate the virtual viewpoint image based on the data selected by the selection unit and viewpoint information indicating a position and a direction of a virtual viewpoint. ### Claim 12 (depends on 1) The apparatus according to claim 1, wherein the components of the three-dimensional shape model are one of points in a case in which the three-dimensional shape model is formed by a point cloud and meshes in a case in which the three-dimensional shape model is formed by a set of meshes. ### Claim 13 (depends on 1) The apparatus according to claim 1, wherein the predetermined storage unit is one of a storage unit included in the information processing apparatus and a storage unit included in an apparatus to be connected to the information processing apparatus. ### Claim 14 (depends on 1) The apparatus according to claim 1, wherein the visibility information related to the components of the three-dimensional shape model is information indicating whether the positions of the components are visible from the positions of the plurality of the cameras. ### Claim 15 A method of controlling an information processing apparatus that performs processing to generate a virtual viewpoint image based on a plurality of captured images obtained by performing image capturing from a plurality of directions by a plurality of cameras, the method comprising: obtaining visibility information which is related to components of a three-dimensional shape model of an object to be captured by at least one of the plurality of cameras, wherein the visibility information is configured to be used for identifying whether pixels corresponding to the components are included in a captured image for each of not less than two captured images of the plurality of captured images; and selecting, based on the obtained visibility information, from material data which is based on the captured image obtained by a camera included in the plurality of cameras and is stored in a predetermined storage unit, data to be used to generate the virtual viewpoint image. ### Claim 16 (depends on 15) The method according to claim 15, wherein the material data stored in the predetermined storage unit includes at least one of partial image data extracted from the captured image and component data of the three-dimensional shape model which is generated based on the plurality of the captured images. ### Claim 17 (depends on 15) The method according to claim 15, further comprising: determining, based on the obtained visibility information, a priority of the material data stored in the predetermined storage unit, wherein the data to be used for the generation of the virtual viewpoint image is selected based on the determined priority. ### Claim 18 A non-transitory computer-readable storage medium that stores a program configured to cause a computer to execute a method of controlling an information processing apparatus that performs processing to generate a virtual viewpoint image based on a plurality of captured images obtained by performing image capturing from a plurality of directions by a plurality of cameras, the method comprising: obtaining visibility information which is related to components of a three-dimensional shape model of an object to be captured by at least one of the plurality of cameras, wherein the visibility information is configured to be used for identifying identify whether pixels corresponding to the components are included in a captured image for each of not less than two captured images of the plurality of captured images; and selecting, based on the obtained visibility information, from material data which is based on the captured image obtained by a camera included in the plurality of cameras and is stored in a predetermined storage unit, data to be used to generate the virtual viewpoint image.