# US 10,872,462 B2 · Image processing apparatus, method, and non-transitory computer-readable medium for rendering a virtual viewpoint image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,462 B2 - Document ID: US10872462B2 - Status: Granted - Kind: B2 - Application number: 16/056,898 - Filed: 2018-08-07 - Published: 2019-02-14 - Granted: 2020-12-22 - Earliest claimed: 2017-08-10 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Tomohiro Nishiyama - Web page: https://patentgrep.com/patent/US10872462B2 ## CPC - G06T 15/205: {Image-based rendering} - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/73: using feature-based methods - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - H04N 13/243: using three or more two-dimensional [2D] image sensors - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There is provided with an image processing apparatus. An image obtaining unit obtains images acquired based on image capturing for a target area from a plurality of directions by a plurality of cameras. An information obtaining unit obtains viewpoint information indicating a position of a virtual viewpoint. A setting unit sets, based on a reference position within the target area and the viewpoint information obtained by the information obtaining unit, a parameter relating to a resolution of an object within the target area. A generating unit generates, based on the images obtained by the image obtaining unit and the viewpoint information obtained by the information obtaining unit, a virtual viewpoint image that includes an image of the object with the resolution according to the parameter set by the setting unit. ## Claims ### Claim 1 An image processing apparatus comprising: one or more memories that store a set of instructions; and one or more processors that execute the set of instructions to: obtain images acquired based on image capturing for a target area from a plurality of directions by a plurality of image capturing devices; obtain viewpoint information indicating a position of a virtual viewpoint; determine, based on a positional relationship between a reference position within the target area and the position of the virtual viewpoint indicated by the obtained viewpoint information, a parameter indicating a size of a constituent element of a three-dimensional model which is data indicating a shape of an object within the target area; generate a three-dimensional model based on the obtained images and the size of the constituent element according to the determined parameter; and generate a virtual viewpoint image based on the generated three-dimensional model and the obtained viewpoint information. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more processors execute the instructions to: determine, based on the obtained viewpoint information, a distance between the reference position and the position of the virtual viewpoint indicated by the obtained viewpoint information, wherein the parameter is determined, based on the determined distance. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more processors execute the instructions to: obtain device information indicating positions of the plurality of image capturing devices, wherein the parameter is determined based on the positional relationship and the obtained device information. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more processors execute the instructions to: accept an instruction for controlling a resolution of a virtual viewpoint image, wherein the parameter is determined based on the positional relationship and the accepted instruction. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the reference position is a position of the object within the target area. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the parameter is determined for each of a plurality of objects, which are a plurality of the reference positions, based on a position of the object and the obtained viewpoint information. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more processors execute the instructions to: select one of a plurality of objects within the target area based on the obtained viewpoint information, wherein the parameter is determined based on a positional relationship between a position of the selected object which is the reference position and the position of the virtual viewpoint indicated by the obtained viewpoint information. ### Claim 8 (depends on 7) The image processing apparatus according to claim 7, wherein the selected object is an object, from among the plurality of objects, closest to the virtual viewpoint indicated by the obtained viewpoint information. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the parameter for each frame image of a plurality of frame images of the virtual viewpoint image is determined, based on a positional relationship between a position of an object for the frame image, which is the reference position, and the position of the virtual viewpoint indicated by the obtained viewpoint information. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the reference position is a position that the plurality of image capturing devices are directed toward. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein information indicating a direction from a virtual viewpoint is included in the viewpoint information. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein the parameter is determined so that resolution parameters of the reference position for a plurality of frame images of a virtual viewpoint image are included within a predetermined range. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the parameter is determined so as to represent a specific object at the reference position using a specific number of pixels in the virtual viewpoint image. ### Claim 14 A method comprising: obtaining images acquired based on image capturing for a target area from a plurality of directions by a plurality of image capturing devices; obtaining viewpoint information indicating a position of a virtual viewpoint; determining, based on a positional relationship between a reference position within the target area and the position of the virtual viewpoint indicated by the obtained viewpoint information, a parameter indicating a size of a constituent element of a three-dimensional model which is data indicating a shape of an object within the target area; generating a three-dimensional model based on the obtained images and the size of the constituent element according to the determined parameter; and generating a virtual viewpoint image based on the generated three-dimensional model and the obtained viewpoint information. ### Claim 15 A non-transitory computer-readable medium storing a computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform: obtaining images acquired based on image capturing for a target area from a plurality of directions by a plurality of image capturing devices; obtaining viewpoint information indicating a position of a virtual viewpoint; determining, based on a positional relationship between a reference position within the target area and the position of the virtual viewpoint indicated by the obtained viewpoint information, a parameter indicating a size of a constituent element of a three-dimensional model which is data indicating a shape of an object within the target area; generating a three-dimensional model based on the obtained images and the size of the constituent element according to the determined parameter; and generating a virtual viewpoint image based on the generated three-dimensional model and the obtained viewpoint information.