# US 2020/0288097 A1 · Method, apparatus, medium, terminal, and device for multi-angle free-perspective interaction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0288097 A1
- Document ID: US20200288097A1
- Status: Published application
- Kind: A1
- Application number: 16/810,352
- Filed: 2020-03-05
- Published: 2020-09-10
- Earliest claimed: 2019-03-07
- Assignees: Alibaba Group Holding Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Xiaojie Sheng
- Web page: https://patentgrep.com/patent/US20200288097A1

## CPC

- G06T 15/205: {Image-based rendering}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 15/04: Texture mapping
- G06T 15/20: Perspective computation
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/128: Adjusting depth or disparity
- H04N 13/139: Format conversion, e.g. of frame-rate or size
- H04N 13/156: Mixing image signals
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/172: image signals comprising non-image signal components, e.g. headers or format information
- H04N 13/189: Recording image signals; Reproducing recorded image signals
- H04N 13/268: based on depth image-based rendering [DIBR]
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/349: Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307)
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2210/36: Level of detail
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, apparatuses, media, terminals, and devices for multi-angle free-perspective interaction are provided. The method for multi-angle free perspective interaction includes: receiving a user instruction; determining a virtual viewpoint according to the user instruction, where the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of a to-be-viewed area; and displaying content for viewing the to-be-viewed area based on the virtual viewpoint, where the content is generated based on a data combination and the virtual viewpoint, and the data combination includes pixel data and depth data of multiple synchronized images, and an association relationship exists between the image data and the depth data of each image, and the multiple synchronized images have different perspectives with respect to the to-be-viewed area.

## Claims

### Claim 1

A method for multi-angle free perspective interaction, comprising:
receiving a user instruction;
determining a virtual viewpoint according to the user instruction, wherein the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of a to-be-viewed area; and
displaying content for viewing the to-be-viewed area based on the virtual viewpoint, wherein the content is generated based on a data combination and the virtual viewpoint, and the data combination includes pixel data and depth data of multiple synchronized images, and an association relationship exists between image data the depth data of a respective image of the multiple synchronized images, and the multiple synchronized images have different perspectives with respect to the to-be-viewed area.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the virtual viewpoint according to the user instruction comprises:
determining a basic viewpoint for viewing the to-be-viewed area, wherein the basic viewpoint includes a position of the basic viewpoint and a perspective of the basic viewpoint; and
determining the virtual viewpoint based on the user instruction and an association relationship between the user instruction and a changing manner of the virtual viewpoint based on the basic viewpoint, with the basic viewpoint as a reference.

### Claim 3 (depends on 2)

The method of claim 2, wherein receiving the user instruction comprises:
detecting a path of a touchpoint on a touch-sensitive screen, wherein the path includes at least one of a start point, an end point, and a moving direction of the touchpoint, with the path as the user instruction.

### Claim 4 (depends on 3)

The method of claim 3, wherein the association relationship between the user instruction and the changing manner of the virtual viewpoint based on the basic viewpoint comprises:
the number of the paths is two, wherein a first touchpoint of a first path moves in a direction away from a second touchpoint of the second path, and a position of the virtual viewpoint moves in a direction close to the to-be-viewed area.

### Claim 5 (depends on 3)

The method of claim 3, wherein the association relationship between the user instruction and the changing manner of the virtual viewpoint based on the basic viewpoint comprises:
the number of the paths is two, wherein a first touchpoint of a first path moves in a direction close to a second touchpoint of the second path, and a position of the virtual viewpoint moves in a direction away from the to-be-viewed area.

### Claim 6 (depends on 3)

The method of claim 3, wherein the association relationship between the user instruction and the changing manner of the virtual viewpoint based on the basic viewpoint comprises:
the number of the path is one, wherein a moving distance of the touchpoint is associated with a magnitude of change of the perspective, and a moving direction of the touchpoint is associated with a direction of change of the perspective.

### Claim 7 (depends on 1)

The method of claim 1, wherein the user instruction includes a voice control instruction.

### Claim 8 (depends on 1)

The method of claim 1, wherein the user instruction includes a selection of a preset viewpoint for viewing the to-be-viewed area.

### Claim 9 (depends on 8)

The method of claim 8, wherein the preset viewpoint is taken as the virtual viewpoint.

### Claim 10 (depends on 1)

The method of claim 1, wherein the user instruction includes a selection of a specific object in the to-be-viewed area by the user.

### Claim 11 (depends on 10)

The method of claim 10, wherein prior to receiving the user instruction, the method further comprises:
determining the specific object in the to-be-viewed area through an image recognition technology; and
providing a selection option of the specific object.

### Claim 12 (depends on 1)

The method of claim 1, wherein the user instruction includes at least one of a position and a perspective of the virtual viewpoint.

### Claim 13 (depends on 1)

The method of claim 1, wherein the user instruction includes attitude change information from at least one of a gyroscope and a gravity sensor.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
receiving a trigger instruction; and
receiving the user instruction in response to the trigger instruction.

### Claim 15 (depends on 1)

The method of claim 1, wherein the user instruction is received during the process of playing a video or displaying an image.

### Claim 16

An apparatus, comprising:
an instruction receiving unit, configured to receive a user instruction;
a virtual viewpoint determining unit, configured to determine a virtual viewpoint according to the user instruction, wherein the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of a to-be-viewed area; and
a display unit, configured to display content for viewing the to-be-viewed area based on the virtual viewpoint, wherein the content is generated based on a data combination and the virtual viewpoint, and the data combination includes pixel data and depth data of multiple synchronized images, and an association relationship exists between image data and depth data of a respective image of the multiple synchronized images, and the multiple synchronized images have different perspectives with respect to the to-be-viewed area.

### Claim 17

A computer-readable storage medium having computer instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform acts comprising:
receiving a user instruction;
determining a virtual viewpoint according to the user instruction, wherein the virtual viewpoint is selected from a multi-angle free-perspective range, and the multi-angle free-perspective range is a range supporting virtual viewpoint switching viewing of a to-be-viewed area; and
displaying content for viewing the to-be-viewed area based on the virtual viewpoint, wherein the content is generated based on a data combination and the virtual viewpoint, and the data combination includes pixel data and depth data of multiple synchronized images, and an association relationship exists between image data the depth data of a respective image of the multiple synchronized images, and the multiple synchronized images have different perspectives with respect to the to-be-viewed area.

### Claim 18 (depends on 17)

The computer-readable storage medium of claim 17, wherein determining the virtual viewpoint according to the user instruction comprises:
determining a basic viewpoint for viewing the to-be-viewed area, wherein the basic viewpoint includes a position of the basic viewpoint and a perspective of the basic viewpoint; and
determining the virtual viewpoint based on the user instruction and an association relationship between the user instruction and a changing manner of the virtual viewpoint based on the basic viewpoint, with the basic viewpoint as a reference.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein receiving the user instruction comprises:
detecting a path of a touchpoint on a touch-sensitive screen, wherein the path includes at least one of a start point, an end point, and a moving direction of the touchpoint, with the path as the user instruction.

### Claim 20 (depends on 17)

The computer-readable storage medium of claim 17, wherein the user instruction includes a voice control instruction.
