# US 11,451,759 B2 · Generation apparatus, generation method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,759 B2 - Document ID: US11451759B2 - Status: Granted - Kind: B2 - Application number: 17/011,579 - Filed: 2020-09-03 - Published: 2020-12-24 - Granted: 2022-09-20 - Earliest claimed: 2018-03-09 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Hiroyasu Ito - Web page: https://patentgrep.com/patent/US11451759B2 ## CPC - H04N 13/189: Recording image signals; Reproducing recorded image signals - G06F 16/116: {Details of conversion of file system types or formats} - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06T 1/00: General purpose image data processing - H04N 1/00204: {with a digital computer or a digital computer system, e.g. an internet server (programmed control between transmitter and receiver or between image input and image output device H04N1/32561)} - H04N 1/2112: {using still video cameras} - H04N 1/2125: {Display of information relating to the still picture recording} - H04N 1/32101: {Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title} - H04N 1/32128: {attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image (for information embedded in the image H04N1/32144)} - H04N 13/178: Metadata, e.g. disparity information - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 23/60: Control of cameras or camera modules - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A generation apparatus according to the present invention is a generation apparatus for generating a media file storing virtual viewpoint image data generated based on pieces of image data of an object captured from a plurality of directions with a plurality of cameras, and obtains a virtual viewpoint parameter to be used to generate virtual viewpoint image data. Further, the generation apparatus generates a media file storing the obtained virtual viewpoint parameter and virtual viewpoint image data generated based on the virtual viewpoint parameter. In this way, the generation apparatus can improve usability related to a virtual viewpoint image. ## Claims ### Claim 1 A generation apparatus comprising: one or more memories; and one or more processors functioning by executing instructions stored in the one or more memories as the following units: an obtaining unit configured to obtain a plurality of sets of virtual viewpoint parameters, the virtual viewpoint parameters indicating a position of a virtual viewpoint and a direction from a virtual viewpoint, and being used to for generating virtual viewpoint image data generated based on image data obtained based on image capturing by a plurality of imaging devices; and a generation unit configured to generate a media file storing the plurality of sets of virtual viewpoint parameters obtained by the obtaining unit and virtual viewpoint image data generated based on the plurality of sets of virtual viewpoint parameters obtained by the obtaining unit, (i) the plurality of sets of virtual viewpoint parameters being stored in a header area in a head of the media file, (ii) each of sets of virtual viewpoint parameters corresponding to one of frames included in the virtual viewpoint image data, and (iii) a set of virtual viewpoint parameters including a parameter for specifying one of the frames included in the virtual viewpoint image data, a parameter indicating a position of the virtual viewpoint corresponding to the one of the frames included in the virtual viewpoint image data, and a parameter indicating a direction from the virtual viewpoint corresponding to the one of the frames included in the virtual viewpoint image data. ### Claim 2 (depends on 1) The generation apparatus according to claim 1, wherein the media file is a file in a still image format, and the generation unit stores the plurality of sets of virtual viewpoint parameters obtained by the obtaining unit in a metadata area in the media file. ### Claim 3 (depends on 2) The generation apparatus according to claim 2, wherein the still image format is Exif, and the plurality of sets of virtual viewpoint parameters obtained by the obtaining unit is stored in an IFD area defined in an Exif standard inside the metadata area in the media file. ### Claim 4 (depends on 2) The generation apparatus according to claim 2, wherein the still image format is Exif, and the plurality of sets of virtual viewpoint parameters obtained by the obtaining unit is stored in an APPn marker segment not defined in an Exif standard inside the metadata area in the media file. ### Claim 5 (depends on 1) The generation apparatus according to claim 1, wherein the media file is a file in a moving image format, and the generation unit stores the virtual viewpoint image data and the plurality of sets of virtual viewpoint parameters in the media file in a chronological order of the frames. ### Claim 6 (depends on 5) The generation apparatus according to claim 5, wherein the moving image format is Ogg Vorbis. ### Claim 7 (depends on 1) The generation apparatus according to claim 1, wherein the media file is a file in a moving image format, and the moving image format is Ogg Vorbis. ### Claim 8 (depends on 1) The generation apparatus according to claim 1, wherein the set of virtual viewpoint parameters further includes a parameter indicating a viewing angle of the virtual viewpoint. ### Claim 9 (depends on 1) The generation apparatus according to claim 1, wherein a virtual viewpoint image displayed on a display screen based on the media file includes a figure representing a trajectory of the virtual viewpoint corresponding to the virtual viewpoint parameter in the media file. ### Claim 10 (depends on 1) The generation apparatus according to claim 1, wherein the parameter for specifying the one of the frames included in the virtual viewpoint image data is a flame number of the one of the frames included in the virtual viewpoint image data. ### Claim 11 (depends on 1) The generation apparatus according to claim 1, wherein the parameter for specifying the one of the frames included in the virtual viewpoint image data is a time code of the one of the frames included in the virtual viewpoint image data. ### Claim 12 A generation method comprising: obtaining a plurality of sets of virtual viewpoint parameters, the virtual viewpoint parameters indicating a virtual viewpoint and a direction from a virtual viewpoint, and being used for generating virtual viewpoint image data generated based on image data obtained based on image capturing by a plurality of imaging devices; and generating a media file storing the plurality of sets of virtual viewpoint parameters and virtual viewpoint image data generated based on the obtained plurality of sets of virtual viewpoint parameters, (i) the plurality of sets of virtual viewpoint parameters being stored in a header area in a head of the media file, (ii) each of sets of virtual viewpoint parameters corresponding to one of frames included in the virtual viewpoint image data, and (iii) a set of virtual viewpoint parameters including a parameter for specifying one of the frames included in the virtual viewpoint image data, a parameter indicating a position of the virtual viewpoint corresponding to the one of the frames included in the virtual viewpoint image data, and a parameter indicating a direction from the virtual viewpoint corresponding to the one of the frames included in the virtual viewpoint image data. ### Claim 13 A non-transitory computer readable storage medium storing a program which causes a computer to perform a generation method, the generation method comprising: obtaining a plurality of sets of virtual viewpoint parameters, the virtual viewpoint parameters indicating a virtual viewpoint and a direction from a virtual viewpoint, and being used for generating virtual viewpoint image data generated based on image data obtained based on image capturing by a plurality of imaging devices; and generating a media file storing the plurality of sets of virtual viewpoint parameters and virtual viewpoint image data generated based on the obtained plurality of sets of virtual viewpoint parameters, (i) the plurality of sets of virtual viewpoint parameters being stored in a header area in a head of the media file, (ii) each of sets of virtual viewpoint parameters corresponding to one of frames included in the virtual viewpoint image data, and (iii) a set of virtual viewpoint parameters including a parameter for specifying one of the frames included in the virtual viewpoint image data, a parameter indicating a position of the virtual viewpoint corresponding to the one of the frames included in the virtual viewpoint image data, and a parameter indicating a direction from the virtual viewpoint corresponding to the one of the frames included in the virtual viewpoint image data.