# US 2019/0356906 A1 · Method for generating virtual viewpoint image and image processing apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0356906 A1 - Document ID: US20190356906A1 - Status: Published application - Kind: A1 - Application number: 16/303,477 - Filed: 2017-05-22 - Published: 2019-11-21 - Earliest claimed: 2016-05-25 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Masahiro Handa; Michio Aizawa; Shogo Mizuno; Katsumasa Tanaka; Akihiro Matsushita; Keisuke Morisawa; Tomohiro Yano; Mai Komiyama; Kenichi Fujii; Atsushi Date - Web page: https://patentgrep.com/patent/US20190356906A1 ## CPC - G06T 15/205: {Image-based rendering} - H04N 13/243: using three or more two-dimensional [2D] image sensors - G06T 7/73: using feature-based methods - H04N 5/2224: {related to virtual studio applications} - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 13/296: Synchronisation thereof; Control thereof - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for generating a virtual viewpoint image includes generating, by a first image processing apparatus, first information for generating the virtual viewpoint image based on a first image captured by a first camera, generating, by a second image processing apparatus, second information for generating the virtual viewpoint image based on a second image captured by a second camera; specifying a viewpoint in the virtual viewpoint image; and generating, by a third image processing apparatus, a virtual viewpoint image corresponding to the specified viewpoint using the first information and the second information. ## Claims ### Claim at position 1 1-15. (canceled) ### Claim 16 An image processing system, comprising: a plurality of cameras configured to capture an imaging target region from a plurality of directions; a plurality of image processing apparatuses including a first image processing apparatus which extracts an object region from a first image captured by a first camera in the plurality of cameras and a second image processing apparatus which extracts an object region from a second image captured by a second camera in the plurality of cameras; and an image generating apparatus configured to generate a virtual viewpoint image based on results of the extraction of the object regions performed by the plurality of image processing apparatuses, wherein the plurality of image processing apparatuses are connected to each other through a daisy chain, and wherein image data based on the results of the extraction of the object regions performed by the first and second image processing apparatuses is transmitted to the image generating apparatus in accordance with the daisy chain connection between the plurality of image processing apparatuses. ### Claim 17 (depends on 16) The image processing system according to claim 16, wherein at least one of the first and second image processing apparatuses extracts a non-object region which is at least different from the object region from the image captured by the corresponding one of the cameras, and wherein at least one of the image processing apparatuses compresses the image data corresponding to the result of the extraction of the object region and image data corresponding to a result of the extraction of the non-object region such that a compression rate of the image data corresponding to the result of the extraction of the object region is lower than a compression rate of the image data corresponding to the result of the extraction of the non-object region and transmits the image data. ### Claim 18 (depends on 16) The image processing system according to claim 16, wherein at least one of the first and second image processing apparatuses extracts a non-object region which is at least different from the object region from the image captured by the corresponding one of the cameras, and wherein at least one of the image processing apparatuses transmits image data corresponding to a result of the extraction of the object region without compression and transmits image data corresponding to a result of the extraction of the non-object region after the image data is compressed. ### Claim 19 (depends on 16) The image processing system according to claim 16, wherein at least one of the first and second image processing apparatuses extracts a non-object region which is at least different from the object region from the image captured by the corresponding one of the cameras, and wherein at least one of the image processing apparatuses transmits image data corresponding to a result of the extraction of the object region and image data corresponding to a result of the extraction of the non-object region such that transmission of the image data corresponding to the result of the extraction of the non-object region is less frequently that transmission of the image data corresponding to the result of the extraction of the object region. ### Claim 20 (depends on 17) The image processing system according to claim 17, wherein the image data corresponding to the result of the extraction of the object region and the image data corresponding to the result of the extraction of the non-object region are transmitted to one of the plurality of image processing apparatuses which is different from the at least one of the image processing apparatuses or the image generating apparatus. ### Claim 21 (depends on 17) The image processing system according to claim 17, wherein the image data corresponding to the result of the extraction of the non-object region is image data of the non-object region. ### Claim 22 (depends on 16) The image processing system according to claim 16, wherein one of the plurality of image processing apparatuses which is different from a first one of the image processing apparatuses in predetermined order generates image data to be transmitted to a next one of the image processing apparatuses in the predetermined order based on image data corresponding to a result of the extraction of the object region and image data corresponding to a result of extraction of an object region performed by a preceding one of the image processing apparatuses in the predetermined order. ### Claim 23 (depends on 16) The image processing system according to claim 16, wherein the image generating apparatus comprising: a reception unit configured to receive information on a virtual viewpoint; and an image generating unit configured to generate the virtual viewpoint image based on a position and an orientation of the virtual viewpoint based on the information received by the reception unit. ### Claim 24 (depends on 23) The image processing system according to claim 23, wherein the image generating apparatus further comprising: a data generating unit configured to generate three-dimensional shape data of objects based on image data corresponding to results of the extraction of the object regions performed by the plurality of image processing apparatuses, and wherein the image generating unit generates a virtual viewpoint image based on the three-dimensional shape data generated by the data generating unit and the position and the orientation of the virtual viewpoint corresponding to the information received by the reception unit. ### Claim 25 (depends on 16) The image processing system according to claim 16, further comprising a synchronization unit configured to synchronize imaging timings of the plurality of cameras. ### Claim 26 (depends on 16) The image processing system according to claim 16, wherein a ratio of the plurality of cameras to the plurality of image processing apparatuses is N to M (N and M are integers not less than 1). ### Claim 27 (depends on 16) The image processing system according to claim 16, wherein the image data corresponding to a result of the extraction of the object region is image data of the object region. ### Claim 28 (depends on 16) The image processing system according to claim 16, wherein the object region includes a region of a moving body. ### Claim 29 (depends on 16) The image processing system according to claim 16, wherein at least one of a region of a person and a region of a ball is included in the object region. ### Claim 30 An image processing apparatus, comprising: an object extraction unit configured to extract an object region from an image captured by a camera; an obtaining unit configured to obtain, from another image processing apparatus, image data corresponding to an object region extracted by the other image processing apparatus from an image captured by another camera; and a transmission unit configured to transmit generation image data based on image data corresponding to a result of the extraction of the object region performed by the object extraction unit and image data obtained from the other image processing apparatus by the obtaining unit to an image generating apparatus which generates a virtual viewpoint image. ### Claim 31 (depends on 30) The image processing apparatus according to claim 30, further comprising: a non-object extraction unit configured to extract a non-object region which is at least different from the object region from an image captured by the camera, wherein the transmission unit compresses the generation image data and image data corresponding to a result of the extraction of the non-object region extracted by the non-object extraction unit such that a compression rate of the generation image data is lower than a compression rate of the image data corresponding to a result of the extraction of the non-object region and transmits the image data. ### Claim 32 (depends on 30) The image processing apparatus according to claim 30, further comprising: a non-object extraction unit configured to extract a non-object region which is at least different from the object region from the image captured by the camera, wherein the transmission unit transmits the generation image data without compression and transmits image data corresponding to a result of the extraction of the non-object region performed by the non-object extraction unit after the image data is compressed. ### Claim 33 (depends on 30) The image processing apparatus according to claim 30, further comprising: a non-object extraction unit configured to extract a non-object region which is at least different from the object region from the image captured by the camera, wherein the transmission unit transmits the generation image data and image data corresponding to a result of the extraction of the non-object region such that transmission of the image data corresponding to the result of the extraction of the non-object region performed by the non-object extraction unit is less frequently than transmission of the generation image data. ### Claim 34 A method for controlling an image processing apparatus, the method comprising: extracting an object region from an image captured by a camera; obtaining, from another image processing apparatus, image data corresponding to an object region extracted by the other image processing apparatus from an image captured by another camera; and transmitting generation image data based on image data corresponding to a result of the extraction of the object region obtained in the extracting and image data obtained from the other image processing apparatus in the obtaining to an image generating apparatus which generates a virtual viewpoint image. ### Claim 35 (canceled)