# US 2019/0174108 A1 · Image processing apparatus, information processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0174108 A1 - Document ID: US20190174108A1 - Status: Published application - Kind: A1 - Application number: 16/202,550 - Filed: 2018-11-28 - Published: 2019-06-06 - Earliest claimed: 2017-12-04 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Tomohiro Yano - Web page: https://patentgrep.com/patent/US20190174108A1 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - G06T 7/55: from multiple images - G06T 15/04: Texture mapping - G06T 15/205: {Image-based rendering} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An information processing apparatus comprises: a capturing information obtaining unit for obtaining capturing information on plural capturing apparatuses for capturing images in plural directions; a model information obtaining unit for obtaining model information on a background shape model indicating a three-dimensional shape of a background of a virtual viewpoint image generated by an image generating unit using the images captured by the capturing apparatuses; and an information generating unit for generating related information associating at least one of the capturing apparatuses with a partial region of the background shape model, based on the obtained capturing information and the obtained model information, wherein the related information is used to determine which one of the capturing apparatuses is to capture the image used by the image generating unit to decide a pixel value corresponding to the partial region in the virtual viewpoint image. ## Claims ### Claim 1 An information processing apparatus comprising: a capturing information obtaining unit configured to obtain capturing information on a plurality of capturing apparatuses for capturing images in a plurality of directions; a model information obtaining unit configured to obtain model information on a background shape model indicating a three-dimensional shape of a background of a virtual viewpoint image that is generated by an image generating unit using the images captured by the plurality of capturing apparatuses; and an information generating unit configured to generate related information that associates at least one of the plurality of capturing apparatuses with a partial region of the background shape model, based on the capturing information obtained by the capturing information obtaining unit and the model information obtained by the model information obtaining unit, wherein the related information is used to determine which one of the plurality of capturing apparatuses is to capture the image used by the image generating unit to decide a pixel value corresponding to the partial region in the virtual viewpoint image. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the information generating unit is configured to select one of the plurality of capturing apparatuses with a capturing range including the partial region of the background shape model, and to generate the related information that associates the selected capturing apparatus with the partial region. ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, wherein the information generating unit is configured to select the capturing apparatus based on a resolution of the partial region in the at least one captured image captured by the at least one capturing apparatus. ### Claim 4 (depends on 2) The information processing apparatus according to claim 2, wherein the information generating unit is configured to select the capturing apparatus based on a size of the background shape model included in the capturing range of the at least one capturing apparatus. ### Claim 5 (depends on 2) The information processing apparatus according to claim 2, wherein the information generating unit is configured to select the capturing apparatus based on a relationship between the at least one capturing apparatus and at least one of a position, a direction, and an exposure value of a reference capturing apparatus decided from among the plurality of capturing apparatuses. ### Claim 6 (depends on 2) The information processing apparatus according to claim 2, wherein the information generating unit is configured to select the capturing apparatus based on a relationship between (i) the at least one capturing apparatus and (ii) at least one of a position and a direction of a virtual viewpoint corresponding to a virtual viewpoint image generated by the image generating unit. ### Claim 7 (depends on 2) The information processing apparatus according to claim 2, wherein the related information generated by the information generating unit indicates a part corresponding to the partial region in an image captured by the selected capturing apparatus. ### Claim 8 (depends on 1) The information processing apparatus according to claim 1, further comprising a control unit configured to input a partial image to the image generating unit, the partial image corresponding to the partial region, the partial image being included in an image captured by the capturing apparatus associated with the partial region according to the related information generated by the information generating unit. ### Claim 9 (depends on 1) The information processing apparatus according to claim 1, wherein the information generating unit is configured to generate the related information again, if a malfunction occurs in one of the capturing apparatuses. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the capturing information obtained by the capturing information obtaining unit indicates positions and directions of the capturing apparatuses. ### Claim 11 (depends on 1) The information processing apparatus according to claim 1, wherein the model information obtained by the model information obtaining unit indicates a position of each of a plurality of partial regions of the background shape model. ### Claim 12 (depends on 1) The information processing apparatus according to claim 1, wherein the image generating unit is configured to generate virtual viewpoint images of a plurality of frames based on a plurality of foreground shape models, indicating foreground three-dimensional shapes in the frames and the background shape model corresponding to the frames. ### Claim 13 (depends on 12) The information processing apparatus according to claim 12, wherein a foreground indicated by the foreground shape model includes a moving object, and a background indicated by the background shape model includes at least one of a ground and a structure. ### Claim 14 An information processing method comprising: obtaining capturing information on a plurality of capturing apparatuses for capturing images in a plurality of directions; obtaining model information on a background shape model indicating a three-dimensional shape of a background of a virtual viewpoint image that is generated by an image generating unit using the images captured by the capturing apparatuses; and generating related information that associates at least one of the capturing apparatuses with a partial region of the background shape model, based on the capturing information and the model information, wherein the related information is used to determine which one of the capturing apparatuses is to capture the image used by the image generating unit to decide a pixel value corresponding to the partial region in the virtual viewpoint image. ### Claim 15 (depends on 14) The information processing method according to claim 14, wherein, in generation of the related information, one of the capturing apparatuses is selected with a capturing range including the partial region of the background shape model, and the related information that associates the selected capturing apparatus with the partial region is generated. ### Claim 16 (depends on 15) The information processing method according to claim 15, wherein the related information indicates a part corresponding to the partial region in an image captured by the selected capturing apparatus. ### Claim 17 A non-transitory computer-readable storage medium that stores a program for causing a computer to perform an information processing method, the information processing method comprising: obtaining capturing information on a plurality of capturing apparatuses for capturing images in a plurality of directions; obtaining model information on a background shape model indicating a three-dimensional shape of a background of a virtual viewpoint image that is generated by an image generating unit using the images captured by the capturing apparatuses; and generating related information that associates at least one of the capturing apparatuses with a partial region of the background shape model, based on the capturing information and the model information, wherein the related information is used to determine which one of the capturing apparatuses is to capture the image used by the image generating unit to decide a pixel value corresponding to the partial region in the virtual viewpoint image.