# US 2020/0288104 A1 · Method, apparatus, medium, and device for generating multi-angle free-respective image data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0288104 A1
- Document ID: US20200288104A1
- Status: Published application
- Kind: A1
- Application number: 16/810,464
- Filed: 2020-03-05
- Published: 2020-09-10
- Earliest claimed: 2019-03-07
- Assignees: Alibaba Group Holding Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Xiaojie Sheng
- Web page: https://patentgrep.com/patent/US20200288104A1

## CPC

- G06T 15/205: {Image-based rendering}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 15/04: Texture mapping
- G06T 15/20: Perspective computation
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/128: Adjusting depth or disparity
- H04N 13/139: Format conversion, e.g. of frame-rate or size
- H04N 13/156: Mixing image signals
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/172: image signals comprising non-image signal components, e.g. headers or format information
- H04N 13/189: Recording image signals; Reproducing recorded image signals
- H04N 13/268: based on depth image-based rendering [DIBR]
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/349: Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307)
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2210/36: Level of detail
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, apparatuses, mediums, and devices for generating multi-angle free-respective image data are provided. The method for generating multi-angle free-perspective image data includes: acquiring multiple synchronized images, where the multiple images have different shooting angles; determining the depth data of each image based on the multiple images; and for each of the images, storing pixel data of the image in a first field and storing depth data in a second field associated with the first field. Technical solutions in the example embodiments of the present invention may improve the user experience.

## Claims

### Claim 1

A method, comprising:
acquiring multiple synchronized images, wherein the multiple synchronized images have different shooting angles;
determining depth data of a respective image of the multiple synchronized images based on the multiple synchronized images; and
for the respective image, storing pixel data of the perspective image in a first field and storing the depth data of the perspective image in a second field associated with the first field.

### Claim 2 (depends on 1)

The method of claim 1, wherein the depth data includes all or some of depth values in a depth value set corresponding to pixel points of the respective image on a one-to-one basis.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the depth data of the respective image includes:
storing a calculated depth value set corresponding to pixel points of the respective image on a one-to-one basis as an original depth map according to an arrangement of the pixel points of the respective image; and
down-sampling the original depth map to obtain the depth data.

### Claim 4 (depends on 1)

The method of claim 1, wherein for the respective image, the pixel data of the respective image is sequentially stored in the first field according to a preset order, and the depth data of the respective image is sequentially stored in the second field according to the preset order.

### Claim 5 (depends on 1)

The method of claim 1, wherein both the first field and the second field are pixel fields in a stitched image, and the stitched image is configured to store the pixel data and the depth data of the multiple synchronized images.

### Claim 6 (depends on 5)

The method of claim 5, wherein the stitched image is split into an image area and a depth map area, and a pixel field of the image area stores the pixel data of the multiple synchronized images, and a pixel field of the depth map area stores depth data of the multiple synchronized images; and
wherein a pixel field storing the pixel data of the respective image in the image area is configured as the first field, and a pixel field storing the depth data of the respective image in the depth map area is configured as the second field.

### Claim 7 (depends on 6)

The method of claim 6, wherein the image area includes multiple image sub-areas, and a respective image sub-area is configured to store one of the multiple synchronized images, and a pixel field of the respective image sub-area is configured as the first field; and
wherein the depth map area includes multiple depth map sub-areas, and a respective depth map sub-area is configured to store depth data of one of the multiple synchronized images, and a pixel field of the respective depth map sub-area is configured as the second field.

### Claim 8 (depends on 7)

The method of claim 7, wherein the pixel data of the respective image is stored in the respective image sub-area in the order of an arrangement of pixel points of the respective image, and the depth data of the respective image is stored in the pixel field of the respective depth map sub-area in the order of an arrangement of pixel points of the respective image.

### Claim 9 (depends on 7)

The method of claim 7, wherein a data amount of the respective depth map sub-area is smaller than a data amount of the respective image sub-area.

### Claim 10 (depends on 7)

The method of claim 7, wherein the pixel field of the respective image sub-area stores three-channel data, and the pixel field of the respective depth map sub-area stores single-channel data.

### Claim 11 (depends on 7)

The method of claim 7, wherein the respective image sub-area stores three-channel data, and the respective depth map sub-area stores dual-channel data.

### Claim 12 (depends on 7)

The method of claim 7, further comprising:
performing padding on all or some of the image sub-areas and the depth map sub-areas.

### Claim 13 (depends on 6)

The method of claim 6, wherein the image area is a continuous area, and the depth map area is a continuous area.

### Claim 14 (depends on 1)

The method of claim 1, wherein storing the pixel data of the respective image in the first field includes: storing data obtained by reducing a resolution of the respective image in the first field.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
generating an association relationship field, wherein the association relationship field indicates an association relationship between the first field and the second field.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
calculating and storing parameter data of the respective image based on the multiple synchronized images, wherein the parameter data includes data of a shooting position and a shooting angle of the respective image.

### Claim 17 (depends on 16)

The method of claim 16, wherein the parameter data further includes internal parameter data, and the internal parameter data includes attribute data of an image capturing device.

### Claim 18 (depends on 1)

The method of claim 1, wherein the multiple synchronized images are multiple synchronized frame images obtained by decoding multiple videos.

### Claim 19

An apparatus, comprising:
a synchronized image acquiring unit, configured to acquire multiple synchronized images, wherein the multiple synchronized images have different shooting angles;
a depth data determining unit, configured to determine depth data of a respective image based on the multiple images; and
a storage unit, configured to, for the respective image, store pixel data of the perspective image in a first field and store the depth data of the perspective image in a second field associated with the first field.

### Claim 20

A computer-readable storage medium having computer instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform acts comprising:
acquiring multiple synchronized images, wherein the multiple synchronized images have different shooting angles;
determining depth data of a respective image of the multiple synchronized images based on the multiple synchronized images; and
for the respective image, storing pixel data of the perspective image in a first field and storing the depth data of the perspective image in a second field associated with the first field.