# US 2021/0344848 A1 · Image processing apparatus, image processing system, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0344848 A1 - Document ID: US20210344848A1 - Status: Published application - Kind: A1 - Application number: 17/373,469 - Filed: 2021-07-12 - Published: 2021-11-04 - Earliest claimed: 2016-10-28 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Yasufumi Takama - Web page: https://patentgrep.com/patent/US20210344848A1 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T 15/20: Perspective computation - H04N 5/2224: {related to virtual studio applications} - H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation} - H04N 13/00: Stereoscopic video systems; Multi-view video systems; Details thereof - H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals - H04N 23/63: by using electronic viewfinders - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image processing system includes an image obtaining unit that obtains images based on capturing from plural directions by plural cameras, an information obtaining unit that obtains viewpoint information indicating a virtual viewpoint, and a generation unit configured to generate virtual viewpoint images on a basis of the obtained images and viewpoint information. The generation unit generates a first virtual viewpoint image outputted to a display apparatus that displays an image for a user to specify a virtual viewpoint and a second virtual viewpoint image outputted to an output destination different from the display apparatus by using at least one of data generated in a process for generating the first virtual viewpoint image by image processing using the plural images obtained by the image obtaining unit and the first virtual viewpoint image, the second virtual viewpoint image having a higher image quality than that of the first virtual viewpoint image. ## Claims ### Claim 1 An image processing system comprising: one or more hardware processors; and one or more memories which store instructions executable by the one or more hardware processors to cause the image processing system to perform at least: obtaining a plurality of images captured from respectively different positions by a plurality of image capturing apparatuses; accepting an input corresponding to an operation for specifying a virtual viewpoint; and generating a plurality of virtual viewpoint images based on the obtained plurality of images and the accepted input, the plurality of virtual viewpoint images including a first virtual viewpoint image and a second virtual viewpoint image, wherein a number of image capturing apparatuses corresponding to an image to be used for generating the second virtual viewpoint image is larger than a number of image capturing apparatuses corresponding to an image to be used for generating the first virtual viewpoint image, and wherein a plurality of image capturing apparatuses corresponding to images to be used for generating the second virtual viewpoint image comprises a plurality of image capturing apparatuses corresponding to images to be used for generating the first virtual viewpoint image. ### Claim 2 (depends on 1) The image processing system according to claim 1, wherein the second virtual viewpoint image has a larger image data size per frame of a moving image than that of the first virtual viewpoint image. ### Claim 3 (depends on 1) The image processing system according to claim 1, wherein the first virtual viewpoint image is a virtual viewpoint image to be broadcast live, and wherein the second virtual viewpoint image is a virtual viewpoint image to be broadcast after recording. ### Claim 4 (depends on 1) The image processing system according to claim 1, wherein the second virtual viewpoint image is generated based on a three dimensional model of an object which is generated based on the obtained plurality of images and whose accuracy is higher than that of a three dimensional model of the object used for generating the first virtual viewpoint image. ### Claim 5 (depends on 1) The image processing system according to claim 1, wherein the second virtual viewpoint image is an image to be displayed for a viewer to see an image generated in response to the operation. ### Claim 6 (depends on 1) The image processing system according to claim 1, wherein the instructions further cause the image processing system to perform outputting the first virtual viewpoint image and the second virtual viewpoint image, wherein timing when the first virtual viewpoint image is output is earlier than timing when the second virtual viewpoint image is output. ### Claim 7 (depends on 1) The image processing system according to claim 1, wherein the instructions further cause the image processing system to perform controlling output of the first virtual viewpoint image and the second virtual viewpoint image so that the first virtual viewpoint image is displayed in a first display area and the second virtual viewpoint image is displayed in a second display area different from the first display area. ### Claim 8 (depends on 1) The image processing system according to claim 1, wherein the second virtual viewpoint image is generated by performing image processing using at least one of image data to be generated in a process of generating the first virtual viewpoint image from the obtained image and the first virtual viewpoint image. ### Claim 9 (depends on 1) The image processing system according to claim 1, wherein the second virtual viewpoint image is generated by performing image processing of increasing an image data size per frame of a moving image on another virtual viewpoint image generated based on the plurality of images and the input, and the first virtual viewpoint image is generated by performing processing that is a part of processing of generating the second virtual viewpoint image from said another virtual viewpoint image and that is executed in a processing time smaller than or equal to a predetermined value. ### Claim 10 (depends on 1) The image processing system according to claim 1, wherein the first virtual viewpoint image is an image indicating a shape of an object to be captured by at least one of the plurality of image capturing apparatuses, and wherein the second virtual viewpoint image is an image indicating a color of the object, the color not appearing in the first virtual viewpoint image, in addition to the shape of the object. ### Claim 11 (depends on 1) The image processing system according to claim 1, wherein a number of gradations of colors included in the second virtual viewpoint image is larger than a number of gradations of colors included in the first virtual viewpoint image. ### Claim 12 (depends on 1) The image processing system according to claim 1, wherein a resolution of the second virtual viewpoint image is higher than a resolution of the first virtual viewpoint image. ### Claim 13 (depends on 1) The image processing system according to claim 1, wherein the instructions further cause the image processing system to perform outputting a parameter related to image quality of a virtual viewpoint image to the generation means, wherein the generation means generates the first virtual viewpoint image and the second virtual viewpoint image based on the parameter related to image quality to be output from the output means. ### Claim 14 (depends on 13) The image processing system according to claim 13, wherein the parameter related to image quality includes at least one of a number of image capturing apparatuses corresponding to a captured image to be used to generate a virtual viewpoint image, a resolution of a virtual viewpoint image, and a permissible time as a processing time for generating a virtual viewpoint image. ### Claim 15 (depends on 1) The image processing system according to claim 1, wherein a processing amount of image processing for generating the second virtual viewpoint image is larger than a processing amount of image processing for generating the first virtual viewpoint image. ### Claim 16 An image processing method comprising: obtaining a plurality of images captured from respectively different positions by a plurality of image capturing apparatuses; accepting an input corresponding to an operation for specifying a virtual viewpoint; and generating a plurality of virtual viewpoint images based on the obtained plurality of images and the accepted input, the plurality of virtual viewpoint images including a first virtual viewpoint image and a second virtual viewpoint image, wherein a number of image capturing apparatuses corresponding to an image to be used for generating the second virtual viewpoint image is larger than a number of image capturing apparatuses corresponding to an image to be used for generating the first virtual viewpoint image, and wherein a plurality of image capturing apparatuses corresponding to images to be used for generating the second virtual viewpoint image comprises a plurality of image capturing apparatuses corresponding to images to be used for generating the first virtual viewpoint image. ### Claim 17 (depends on 16) The image processing method according to claim 16, wherein the second virtual viewpoint image has a larger image data size per frame of a moving image than that of the first virtual viewpoint image. ### Claim 18 (depends on 16) The image processing method according to claim 16, wherein the first virtual viewpoint image is a virtual viewpoint image to be broadcast live, and wherein the second virtual viewpoint image is a virtual viewpoint image to be broadcast after recording. ### Claim 19 (depends on 16) The image processing method according to claim 16, wherein the first virtual viewpoint image is an image indicating a shape of an object to be captured by at least one of the plurality of image capturing apparatuses, and wherein the second virtual viewpoint image is an image indicating a color of the object, the color not appearing in the first virtual viewpoint image, in addition to the shape of the object. ### Claim 20 A non-transitory storage medium that stores a program for causing a computer to execute an image processing method, the image processing method comprising: obtaining a plurality of images captured from respectively different positions by a plurality of image capturing apparatuses; accepting an input corresponding to an operation for specifying a virtual viewpoint; and generating a plurality of virtual viewpoint images based on the obtained plurality of images and the accepted input, the plurality of virtual viewpoint images including a first virtual viewpoint image and a second virtual viewpoint image, wherein a number of image capturing apparatuses corresponding to an image to be used for generating the second virtual viewpoint image is larger than a number of image capturing apparatuses corresponding to an image to be used for generating the first virtual viewpoint image, and wherein a plurality of image capturing apparatuses corresponding to images to be used for generating the second virtual viewpoint image comprises a plurality of image capturing apparatuses corresponding to images to be used for generating the first virtual viewpoint image.