# US 2020/0288103 A1 · Method, apparatus, medium, terminal, and device for processing multi-angle free-perspective data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288103 A1 - Document ID: US20200288103A1 - Status: Published application - Kind: A1 - Application number: 16/810,237 - Filed: 2020-03-05 - Published: 2020-09-10 - Earliest claimed: 2019-03-07 - Assignees: Alibaba Group Holging Limited - Applicants: Alibaba Group Holding Limited - Inventors: Xiaojie Sheng - Web page: https://patentgrep.com/patent/US20200288103A1 ## CPC - G06T 15/205: {Image-based rendering} - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 15/04: Texture mapping - G06T 15/20: Perspective computation - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/128: Adjusting depth or disparity - H04N 13/139: Format conversion, e.g. of frame-rate or size - H04N 13/156: Mixing image signals - H04N 13/167: Synchronising or controlling image signals - H04N 13/172: image signals comprising non-image signal components, e.g. headers or format information - H04N 13/189: Recording image signals; Reproducing recorded image signals - H04N 13/268: based on depth image-based rendering [DIBR] - H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background - H04N 13/296: Synchronisation thereof; Control thereof - H04N 13/349: Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307) - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - H04N 21/816: {involving special video data, e.g 3D video} - H04N 23/60: Control of cameras or camera modules - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2210/36: Level of detail - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method, an apparatus, a medium, a terminal, and a device for processing multi-angle free-perspective data are disclosed. The method includes: acquiring a data header file; determining a defined format of a data file according to a parsing result of the data header file; reading and obtaining a data combination from the data file based on the defined format, where the data combination includes pixel data and depth data of multiple synchronized images, and the multiple synchronized images have different perspectives with respect a to-be-viewed area, and pixel data and depth data of each image of the multiple synchronized images has an association relationship; and performing image or video reconstruction of a virtual viewpoint according to the read data combination, where the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of the to-be-viewed area. ## Claims ### Claim 1 A method acquiring a data header file; determining a defined format of a data file according to a parsing result of the data header file; reading and obtaining a data combination from the data file based on the defined format, wherein the data combination includes pixel data and depth data of multiple synchronized images, and the multiple synchronized images have different perspectives with respect a to-be-viewed area, and pixel data and depth data of each image of the multiple synchronized images has an association relationship; and performing image or video reconstruction of a virtual viewpoint according to the read data combination, wherein the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of the to-be-viewed area. ### Claim 2 (depends on 1) The method according to claim 1, wherein the defined format includes a storage format of the data combination. ### Claim 3 (depends on 2) The method according to claim 2, wherein the storage format of the data combination is a video format. ### Claim 4 (depends on 3) The method according to claim 3, wherein the data combination comprises multiple data combinations, each corresponding to a different frame moment. ### Claim 5 (depends on 2) The method according to claim 2, wherein the storage format of the data combination is an image format. ### Claim 6 (depends on 5) The method according to claim 5, wherein the defined format includes content storage rules of the data combination, and the content storage rules include storage rules for the pixel data and depth data of the multiple synchronized images. ### Claim 7 (depends on 6) The method according to claim 6, wherein the storage rules for the pixel data and depth data of the multiple synchronized images includes storage positions of the pixel data and depth data of the multiple synchronized images in a stitched image, the stitched image is an image or a frame image read from the data file according to an image format or a video format in the defined format. ### Claim 8 (depends on 7) The method according to claim 7, wherein the pixel data of each image in the multiple synchronized images is stored in an image area, and the depth data is stored in a depth area. Storage positions of the pixel data and the depth data of each image in the multiple synchronized images are respectively indicated through the image area and the depth data. ### Claim 9 (depends on 7) The method according to claim 7, wherein the pixel data of each image in the multiple synchronized images is stored in an image sub-area, and the depth data of each image in the multiple synchronized images is stored in a depth sub-area. Storage positions of the pixel data and the depth data of each image in the multiple synchronized images are respectively indicated through the image sub-area and the depth sub-area. ### Claim 10 (depends on 9) The method according to claim 9, wherein the storage rules for the pixel data and depth data of the multiple synchronized images further includes performing padding on all or some parts of the image sub-area and the depth map sub-area. ### Claim 11 (depends on 6) The method according to claim 6, wherein the storage rules for the pixel data and depth data of the multiple synchronized images further includes resolutions of the pixel data and the depth data of the images being identical. ### Claim 12 (depends on 9) The method according to claim 9, wherein the data combination includes a first field storing the pixel data of each image in the multiple synchronized images, and a second field that is associated with the first field and stores the depth data of the respective image. ### Claim 13 (depends on 12) The method according to claim 12, wherein the defined format includes an association relationship between the first field and the second field. ### Claim 14 (depends on 1) The method according to claim 1, wherein the data file further includes parameter data of each image of the multiple synchronized images, the parameter data including data of shooting position and shooting angle of the respective image. ### Claim 15 (depends on 14) The method according to claim 14, wherein the parameter data further includes internal parameter data, the internal parameter data including property data of a capturing device associated with the respective image. ### Claim 16 (depends on 14) The method according to claim 14, wherein the defined format further includes a storage address of the parameter data. ### Claim 17 (depends on 14) The method according to claim 14, wherein performing the image or video reconstruction of the virtual viewpoint according to the read data combination includes: reconstructing an image or a video of the virtual viewpoint according to the data combination based on a relationship between the virtual viewpoint and the parameter data. ### Claim 18 (depends on 1) The method according to claim 1, wherein the defined format further includes storage addresses of the pixel data and the depth data of the multiple synchronized images. ### Claim 19 An apparatus comprising: one or more processors; memory; a data header file acquiring unit stored in the memory and executable by the one or more processors to acquire a data header file; a data header file parsing unit, adapted to determine a defined format of a data file according to a parsing result of the data header file; a data combination acquiring unit stored in the memory and executable by the one or more processors to read and obtain a data combination from the data file based on the defined format, wherein the data combination includes pixel data and depth data of multiple synchronized images, and the multiple synchronized images have different perspectives with respect a to-be-viewed area, and pixel data and depth data of each image of the multiple synchronized images has an association relationship; and a reconstruction unit stored in the memory and executable by the one or more processors to perform image or video reconstruction of a virtual viewpoint according to the read data combination, where the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of the to-be-viewed area. ### Claim 20 One or more computer-readable storage media storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising: acquiring a data header file; determining a defined format of a data file according to a parsing result of the data header file; reading and obtaining a data combination from the data file based on the defined format, wherein the data combination includes pixel data and depth data of multiple synchronized images, and the multiple synchronized images have different perspectives with respect a to-be-viewed area, and pixel data and depth data of each image of the multiple synchronized images has an association relationship; and performing image or video reconstruction of a virtual viewpoint according to the read data combination, wherein the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of the to-be-viewed area.