# US 11,443,478 B2 · Generation apparatus, system and method for generating virtual viewpoint image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,478 B2 - Document ID: US11443478B2 - Status: Granted - Kind: B2 - Application number: 16/930,092 - Filed: 2020-07-15 - Published: 2020-11-05 - Granted: 2022-09-13 - Earliest claimed: 2017-12-14 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Naoki Umemura - Web page: https://patentgrep.com/patent/US11443478B2 ## CPC - G06T 15/20: Perspective computation - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - G06T 7/55: from multiple images - H04N 7/181: {for receiving images from a plurality of remote sources} - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N 23/95: Computational photography systems, e.g. light-field imaging systems - G06T 2207/30196: Human being; Person - G06T 2207/30224: Ball; Puck - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract To make it possible to obtain a natural virtual viewpoint image in which a structure or the like existing within an image capturing scene is represented three-dimensionally so as to be the same as a real one while suppressing a network load at the time of transmission of multi-viewpoint image data. The generation device according to the present invention generates a virtual viewpoint image based on three-dimensional shape data corresponding to an object, three-dimensional shape data corresponding to a structure, background data corresponding to a background different at least from the object and the structure, and information indicating a virtual viewpoint. ## Claims ### Claim 1 A system comprising: one or more memories storing instructions; and one or more processors configured to execute the instructions to perform operations including: generating three-dimensional shape data corresponding to a moving object captured from a plurality of directions, acquiring three-dimensional shape data corresponding to a structure which is different from the moving object and which is captured from the plurality of directions, acquiring background data corresponding to a background which is different at least from the moving object and the structure and which is captured from the plurality of directions, acquiring information representing a virtual viewpoint, and generating an image based on the generated three-dimensional shape data corresponding to the moving object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired virtual viewpoint information, wherein the generated three-dimensional shape data corresponding to the moving object is generated based on a difference between (A) an image acquired based on first image capturing in a state where the structure and the moving object exist and (B) an image acquired based on second image capturing from the same direction as that of the first image capturing in a state where the structure exist and the moving object does not exist, where a timing which the second image capturing was performed is different from a timing which the first image capturing was performed. ### Claim 2 (depends on 1) The system according to claim 1, wherein the acquired three-dimensional shape data corresponding to the structure is generated based on a captured image acquired by image capturing before starting an event where imaging from the plurality of directions is to be performed. ### Claim 3 (depends on 2) The system according to claim 2, wherein the captured image acquired by image capturing before starting the event is (B) the image acquired based on the second image capturing from the same direction as that of the first image capturing in the state where the structure exist and the moving object does not exist. ### Claim 4 (depends on 1) The system according to claim 1, wherein the acquired three-dimensional shape data corresponding to the structure is acquired by generating based on a plurality of images, wherein the plurality of images are based on a plurality of captured images acquired by image capturing from the plurality of directions before starting an event, and wherein an area of the structure is represented by the plurality of images and is distinguished from other areas. ### Claim 5 (depends on 1) The system according to claim 1, wherein the acquired three-dimensional shape data corresponding to the structure is acquired before starting an event. ### Claim 6 (depends on 1) The system according to claim 1, wherein the generated three-dimensional shape data corresponding to the moving object is generated based on (C) a plurality of captured images acquired by image capturing from the plurality of directions after starting an event. ### Claim 7 (depends on 6) The system according to claim 6, wherein each of (C) the plurality of captured images is a captured image including the moving object, the structure, and the background. ### Claim 8 (depends on 1) The system according to claim 1, wherein the generated three-dimensional shape data corresponding to the moving object is generated based on (C) a plurality of captured images acquired by image capturing from the plurality of directions after starting an event and (D) a plurality of captured images acquired by image capturing from the plurality of directions before starting the event. ### Claim 9 (depends on 1) The system according to claim 1, further comprising a communication path, wherein the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired virtual viewpoint information are acquired through the communication path included in the system. ### Claim 10 (depends on 1) The system according to claim 1, wherein the moving object is a person. ### Claim 11 (depends on 1) The system according to claim 1, wherein the moving object is at least one of a person or a ball. ### Claim 12 (depends on 1) The system according to claim 1, wherein the structure is an object whose still state continues. ### Claim 13 (depends on 1) The system according to claim 1, wherein the structure is at least one of a soccer goal configured to be used in a soccer game or a corner flag configured to be used in the soccer game. ### Claim 14 (depends on 1) The system according to claim 1, wherein the structure is an object installed at a predetermined position. ### Claim 15 (depends on 1) The system according to claim 1, wherein the structure is a specified object. ### Claim 16 (depends on 1) The system according to claim 1, wherein at least a part of the structure is configured to be installed on a field having an arrangement configured to receive a person, as the moving object, to play a game. ### Claim 17 (depends on 1) The system according to claim 1, wherein the acquired three-dimensional shape data corresponding to the structure is generated based on a difference between (E) the image acquired based on third image capturing in a state where the structure does not exist and the moving object does not exist and (B) the image acquired based on the second image capturing from the same direction as that of the third image capturing in the state where the structure exist and the moving object does not exist, where the timing which the second image capturing was performed is different from a timing which the third image capturing was performed. ### Claim 18 A method for a system, the method comprising: generating three-dimensional shape data corresponding to a moving object captured from a plurality of directions; acquiring three-dimensional shape data corresponding to a structure which is different from the moving object and which is captured from the plurality of directions; acquiring background data corresponding to a background which is different at least from the moving object and the structure and which is captured from the plurality of directions; acquiring information representing a virtual viewpoint; and generating an image based on the generated three-dimensional shape data corresponding to the moving object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired virtual viewpoint information, wherein the generated three-dimensional shape data corresponding to the moving object is generated based on a difference between (A) an image acquired based on first image capturing in a state where the structure and the moving object exist and (B) an image acquired based on second image capturing from the same direction as that of the first image capturing in a state where the structure exist and the moving object does not exist, where a timing which the second image capturing was performed is different from a timing which the first image capturing was performed. ### Claim 19 A non-transitory computer readable storage medium storing a program to cause a computer to perform a method for a system, the method comprising: generating three-dimensional shape data corresponding to a moving object captured from a plurality of directions; acquiring three-dimensional shape data corresponding to a structure which is different from the moving object and which is captured from the plurality of directions; acquiring background data corresponding to a background which is different at least from the moving object and the structure and which is captured from the plurality of directions; acquiring information representing a virtual viewpoint; and generating an image based on the generated three-dimensional shape data corresponding to the moving object, the acquired three-dimensional shape data corresponding to the structure, the acquired background data, and the acquired virtual viewpoint information, wherein the generated three-dimensional shape data corresponding to the moving object is generated based on a difference between (A) an image acquired based on first image capturing in a state where the structure and the moving object exist and (B) an image acquired based on second image capturing from the same direction as that of the first image capturing in a state where the structure exist and the moving object does not exist, where a timing which the second image capturing was performed is different from a timing which the first image capturing was performed.