# US 2020/0288112 A1 · Method, apparatus, medium, and device for processing multi-angle free-perspective video data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0288112 A1
- Document ID: US20200288112A1
- Status: Published application
- Kind: A1
- Application number: 16/810,681
- Filed: 2020-03-05
- Published: 2020-09-10
- Earliest claimed: 2019-03-07
- Assignees: Alibaba Group Holding Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Xiaojie Sheng
- Web page: https://patentgrep.com/patent/US20200288112A1

## CPC

- G06T 15/205: {Image-based rendering}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 15/04: Texture mapping
- G06T 15/20: Perspective computation
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/128: Adjusting depth or disparity
- H04N 13/139: Format conversion, e.g. of frame-rate or size
- H04N 13/156: Mixing image signals
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/172: image signals comprising non-image signal components, e.g. headers or format information
- H04N 13/189: Recording image signals; Reproducing recorded image signals
- H04N 13/268: based on depth image-based rendering [DIBR]
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/349: Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307)
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2210/36: Level of detail
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method, an apparatus, a medium, and a device for processing multi-angle free-perspective video data are disclosed. The method includes: parsing acquired video data to obtain multiple data combinations of different frame moments, the data combinations including pixel data and depth data of multiple synchronized images, and the multiple synchronized images having different perspectives on a to-be-viewed area; and reconstructing an image associated with a respective virtual viewpoint based on the respective data combination for each frame moment, the respective virtual viewpoint being selected from a multi-angle free-perspective range, and the multi-angle free-perspective range being a range that supports viewing and switching of viewpoints on the to-be-viewed area. The technical solutions in the embodiments of the present disclosure can support range being that supports viewing and switching of viewpoints within a multi-angle free-perspective range.

## Claims

### Claim 1

A method implemented by one or more computing devices, the method comprising:
parsing acquired video data to obtain multiple data combinations of different frame moments, the data combinations including pixel data and depth data of multiple synchronized images, and the multiple synchronized images having different perspectives on a to-be-viewed area; and
reconstructing an image associated with a respective virtual viewpoint based on the respective data combination for each frame moment, the respective virtual viewpoint being selected from a multi-angle free-perspective range, and the multi-angle free-perspective range being a range that supports viewing and switching of viewpoints on the to-be-viewed area.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the depth data of each image of the synchronized images is a set of depth values that are in one-to-one correspondence with pixels of the respective image.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the depth data of each image of the synchronized images is data obtained after up-sampling a depth map, the depth map is an image formed by a set of depth values that are in one-to-one correspondence with pixels of the respective image and arranged according to pixel points of the respective image.

### Claim 4 (depends on 3)

The method according to claim 3, wherein reconstructing the image associated with the respective virtual viewpoint based on the respective data combination comprises:
up-sampling the depth data to obtain the set of depth values that are in one-to-one correspondence with the pixels of the image; and
reconstructing the image associated with the virtual viewpoint according to the pixel data of the synchronized multiple images and the set of depth values.

### Claim 5 (depends on 1)

The method according to claim 1, wherein reconstructing the image associated with the respective virtual viewpoint based on the respective data combination comprises:
determining parameter data of each of the multiple synchronized images, the parameter data including data of a shooting position and a shooting angle of the respective image;
determining parameter data of the virtual viewpoint, the parameter data of the virtual viewpoint including a virtual viewing position and a virtual viewing angle;
determining a plurality of target images among the multiple synchronized images;
mapping, for each target image, depth data to the virtual viewpoint according to a relationship between the parameter data of the virtual viewpoint and parameter data of the image; and
generating a reconstructed image according to the depth data mapped to the virtual viewpoint and pixel data of the target image.

### Claim 6 (depends on 5)

The method according to claim 5, wherein determining the plurality of target images among the multiple synchronized images comprises: selecting a target image from among the multiple images according to the relationship between the parameter data of the virtual viewpoint and parameter data of the image.

### Claim 7 (depends on 6)

The method according to claim 6, wherein all of the multiple synchronized images are treated as the target images.

### Claim 8 (depends on 6)

The method according to claim 6, wherein parameters of the image further comprises internal parameter data, the internal parameter data including property data of a shooting device of the image.

### Claim 9 (depends on 1)

The method according to claim 1, wherein: prior to reconstructing the image associated with the virtual viewpoint, the method further comprises: receiving parameter data of the virtual viewpoint.

### Claim 10 (depends on 9)

The method according to claim 9, wherein after reconstructing the image associated with the virtual viewpoint, the method further comprises: sending the reconstructed image to an image display device.

### Claim 11

One or more computer readable media storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:
parsing acquired video data to obtain multiple data combinations of different frame moments, the data combinations including pixel data and depth data of multiple synchronized images, and the multiple synchronized images having different perspectives on a to-be-viewed area; and
reconstructing an image associated with a respective virtual viewpoint based on the respective data combination for each frame moment, the respective virtual viewpoint being selected from a multi-angle free-perspective range, and the multi-angle free-perspective range being a range that supports viewing and switching of viewpoints on the to-be-viewed area.

### Claim 12 (depends on 11)

The one or more computer readable media according to claim 11, wherein the depth data of each image of the synchronized images is a set of depth values that are in one-to-one correspondence with pixels of the respective image.

### Claim 13 (depends on 11)

The one or more computer readable media according to claim 11, wherein the depth data of each image of the synchronized images is data obtained after up-sampling a depth map, the depth map is an image formed by a set of depth values that are in one-to-one correspondence with pixels of the respective image and arranged according to pixel points of the respective image.

### Claim 14 (depends on 13)

The one or more computer readable media according to claim 13, wherein reconstructing the image associated with the respective virtual viewpoint based on the respective data combination comprises:
up-sampling the depth data to obtain the set of depth values that are in one-to-one correspondence with the pixels of the image; and
reconstructing the image associated with the virtual viewpoint according to the pixel data of the synchronized multiple images and the set of depth values.

### Claim 15 (depends on 11)

The one or more computer readable media according to claim 11, wherein reconstructing the image associated with the respective virtual viewpoint based on the respective data combination comprises:
determining parameter data of each of the multiple synchronized images, the parameter data including data of a shooting position and a shooting angle of the respective image;
determining parameter data of the virtual viewpoint, the parameter data of the virtual viewpoint including a virtual viewing position and a virtual viewing angle;
determining a plurality of target images among the multiple synchronized images;
mapping, for each target image, depth data to the virtual viewpoint according to a relationship between the parameter data of the virtual viewpoint and parameter data of the image; and
generating a reconstructed image according to the depth data mapped to the virtual viewpoint and pixel data of the target image.

### Claim 16 (depends on 15)

The one or more computer readable media according to claim 15, wherein determining the plurality of target images among the multiple synchronized images comprises: selecting a target image from among the multiple images according to the relationship between the parameter data of the virtual viewpoint and parameter data of the image.

### Claim 17 (depends on 16)

The one or more computer readable media according to claim 16, wherein parameters of the image further comprises internal parameter data, the internal parameter data including property data of a shooting device of the image.

### Claim 18 (depends on 11)

The one or more computer readable media according to claim 11, wherein: prior to reconstructing the image associated with the virtual viewpoint, the acts further comprise: receiving parameter data of the virtual viewpoint.

### Claim 19 (depends on 18)

The one or more computer readable media according to claim 18, wherein after reconstructing the image associated with the virtual viewpoint, the acts further comprise: sending the reconstructed image to an image display device.

### Claim 20

An apparatus comprising:
one or more processors;
memory;
a parsing unit stored in the memory and executable by the one or more processors to parse acquired video data to obtain multiple data combinations of different frame moments, the data combinations including pixel data and depth data of multiple synchronized images, and the multiple synchronized images having different perspectives on a to-be-viewed area; and
a virtual viewpoint image reconstruction unit stored in the memory and executable by the one or more processors to reconstruct an image associated with a respective virtual viewpoint based on the respective data combination for each frame moment, the respective virtual viewpoint being selected from a multi-angle free-perspective range, and the multi-angle free-perspective range being a range that supports viewing and switching of viewpoints on the to-be-viewed area.
