# US 2020/0404327 A1 · Multi-viewport transcoding for volumetric video streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404327 A1
- Document ID: US20200404327A1
- Status: Published application
- Kind: A1
- Application number: 16/447,144
- Filed: 2019-06-20
- Published: 2020-12-24
- Earliest claimed: 2019-06-20
- Assignees: Indiana University Research and Technology Corporation
- Applicants: AT&T Intellectual Property I, L.P.
- Inventors: Bo Han; Jackson Jarrell Pair; Vijay Gopalakrishnan; Feng Qian
- Web page: https://patentgrep.com/patent/US20200404327A1

## CPC

- H04N 13/388: Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume
- H04N 19/597: specially adapted for multi-view video sequence encoding
- G06T 7/73: using feature-based methods
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 19/162: User input
- H04N 19/172: the region being a picture, frame or field
- H04N 19/40: using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- H04N 13/194: Transmission of image signals
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In one example, a processing system including at least one processor may obtain at least one frame of a volumetric video, obtain a first viewport of a client device, render a plurality of two-dimensional subframes comprising two-dimensional projections of the at least one frame of the volumetric video, based upon the first viewport of the client device, and transmit the plurality of two-dimensional subframes to the client device. In another example, a processing system including at least one processor may request a volumetric video from a network-based proxy, provide a viewport to the network-based proxy, obtain a plurality of two-dimensional subframes from the network-based proxy in accordance with the viewport, select at least one of the plurality of two-dimensional subframes to present, and display the at least one two-dimensional subframe that is selected.

## Claims

### Claim 1

A method comprising:
obtaining, by a processing system including at least one processor, at least one frame of a volumetric video;
obtaining, by the processing system, a first viewport of a client device;
rendering, by the processing system, a plurality of two-dimensional subframes comprising two-dimensional projections of the at least one frame of the volumetric video, based upon the first viewport of the client device; and
transmitting, by the processing system, the plurality of two-dimensional subframes to the client device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the volumetric video is encoded as a compressed volumetric video, wherein the method further comprises:
decoding the compressed volumetric video.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first viewport comprises a position and an orientation.

### Claim 4 (depends on 3)

The method of claim 3, wherein the orientation comprises a yaw, a pitch, and a roll.

### Claim 5 (depends on 3)

The method of claim 3, wherein the first viewport further comprises a field-of-view.

### Claim 6 (depends on 5)

The method of claim 5, wherein at least one of the plurality of two-dimensional subframes is rendered to include visual information that exceeds the field-of-view using a panoramic representation.

### Claim 7 (depends on 1)

The method of claim 1, wherein the plurality of two-dimensional subframes comprises:
a first two-dimensional subframe rendered from the first viewport; and
a second two-dimensional subframe rendered from a second viewport, wherein the second viewport is offset from the first viewport with respect to at least one of:
a position within a space of the volumetric video; or
an orientation within the space of the volumetric video.

### Claim 8 (depends on 7)

The method of claim 7, wherein the first viewport comprises a predicted viewport.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first two-dimensional subframe and the second two-dimensional subframe are rendered as two-dimensional projections from a same frame of the at least one frame of the volumetric video.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
encoding the plurality of two-dimensional subframes, wherein the transmitting comprises transmitting to the client device the plurality of two-dimensional subframes that is encoded.

### Claim 11 (depends on 10)

The method of claim 10, wherein the encoding comprises an interframe compression.

### Claim 12 (depends on 11)

The method of claim 11, wherein the interframe compression is among time-sequential subframes of the plurality of two-dimensional subframes.

### Claim 13 (depends on 12)

The method of claim 12, wherein the interframe compression is among at least one of:
spatial-adjacent subframes of the plurality of two-dimensional subframes; or
spatial-overlapping subframes of the plurality of two-dimensional subframes.

### Claim 14 (depends on 1)

The method of claim 1, wherein the first viewport of the client device is obtained from the client device.

### Claim 15 (depends on 1)

The method of claim 1, wherein the rendering the plurality of two-dimensional subframes based upon the first viewport of the client device comprises:
generating a predicted viewport based upon at least the first viewport in accordance with a machine learning model.

### Claim 16 (depends on 15)

The method of claim 15, wherein the machine learning model generates the predicted viewport based upon a plurality of prior viewports, the plurality of prior viewports including the first viewport.

### Claim 17 (depends on 1)

The method of claim 1, wherein the at least one frame of the volumetric video comprises a point cloud.

### Claim 18 (depends on 1)

The method of claim 1, wherein the at least one frame of the volumetric video comprises a three-dimensional mesh.

### Claim 19

A device comprising:
a processor; and
a computer-readable medium storing instructions which, when executed by the processor, cause the processor to perform operations, the operations comprising:
obtaining at least one frame of a volumetric video;
obtaining a first viewport of a client device;
rendering a plurality of two-dimensional subframes comprising two-dimensional projections of the at least one frame of the volumetric video, based upon the first viewport of the client device; and
transmitting the plurality of two-dimensional subframes to the client device.

### Claim 20

A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising:
obtaining at least one frame of a volumetric video;
obtaining a first viewport of a client device;
rendering a plurality of two-dimensional subframes comprising two-dimensional projections of the at least one frame of the volumetric video, based upon the first viewport of the client device; and
transmitting the plurality of two-dimensional subframes to the client device.
