# US 10,484,682 B2 · Reference picture derivation and motion compensation for 360-degree video coding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,484,682 B2
- Document ID: US10484682B2
- Status: Granted
- Kind: B2
- Application number: 16/025,729
- Filed: 2018-07-02
- Published: 2019-01-03
- Granted: 2019-11-19
- Earliest claimed: 2017-07-03
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Muhammed Zeyd Coban; Geert Van der Auwera; Marta Karczewicz
- Web page: https://patentgrep.com/patent/US10484682B2

## CPC

- H04N 19/115: Selection of the code volume for a coding unit prior to coding
- G06T 15/205: {Image-based rendering}
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/134: characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/51: Motion estimation or motion compensation
- H04N 19/563: Motion estimation with padding, i.e. with filling of non-object values in an arbitrarily shaped picture block or region for estimation purposes
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- H04N 19/88: involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
- H04N 19/96: Tree coding, e.g. quad-tree coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This disclosure describes techniques for generating reference frames packed with extended faces from a cubemap projection or adjusted cubemap projection of 360-degree video data. The reference frames packed with the extended faces may be used for inter-prediction of subsequent frames of 360-degree video data.

## Claims

### Claim 1

A method of decoding 360-degree video data, the method comprising:
receiving an encoded frame of 360-degree video data, the encoded frame of 360-degree video data being arranged in packed faces obtained from a projection of a sphere of the 360-degree video data;
decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data being arranged in the packed faces;
deriving a decoded sphere of 360-degree video data from the decoded frame of 360-degree video data;
sampling the decoded sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the decoded frame of 360-degree video data;
deriving an extended reference frame from the extended faces; and
decoding a subsequent encoded frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 2 (depends on 1)

The method of claim 1, wherein the projection is a cubemap projection or an adjusted cubemap projections (ACP).

### Claim 3 (depends on 1)

The method of claim 1, wherein sampling the decoded sphere of 360-degree video data to produce extended faces using the projection comprises sampling the decoded sphere of 360-degree video data to produce extended faces using the projection in accordance with a number of extension pixels.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
receiving an indication of the number of extension pixels at a picture level or a sequence level.

### Claim 5 (depends on 3)

The method of claim 3, wherein the number of extension pixels is equal to a largest prediction unit size in a coding tree unit (CTU) of the encoded frame of 360-degree video data.

### Claim 6 (depends on 1)

The method of claim 1, wherein decoding the subsequent encoded frame of 360-degree video data using the inter-prediction process and the derived extended reference frame comprises:
rotating a current prediction unit in a current cube face of the encoded frame of 360-degree video data based on a rotation angle of a reference cube face containing a reference block for the current prediction unit.

### Claim 7

An apparatus configured to decode 360-degree video data, the apparatus comprising:
a memory configured to store an encoded frame of 360-degree video data; and
one or more processors in communication with the memory, the one or more processors configured to:
receive the encoded frame of 360-degree video data, the encoded frame of 360-degree video data being arranged in packed faces obtained from a projection of a sphere of the 360-degree video data;
decode the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data being arranged in the packed faces;
derive a decoded sphere of 360-degree video data from the decoded frame of 360-degree video data;
sample the decoded sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the decoded frame of 360-degree video data;
derive an extended reference frame from the extended faces; and
decode a subsequent encoded frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the projection is a cubemap projection or an adjusted cubemap projections (ACP).

### Claim 9 (depends on 7)

The apparatus of claim 7, wherein to sample the decoded sphere of 360-degree video data to produce extended faces using the projection, the one or more processors are further configured to sample the decoded sphere of 360-degree video data to produce extended faces using the projection in accordance with a number of extension pixels.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the one or more processors are further configured to:
receive an indication of the number of extension pixels at a picture level or a sequence level.

### Claim 11 (depends on 9)

The apparatus of claim 9, wherein the number of extension pixels is equal to a largest prediction unit size in a coding tree unit (CTU) of the encoded frame of 360-degree video data.

### Claim 12 (depends on 7)

The apparatus of claim 7, wherein to decode the subsequent encoded frame of 360-degree video data using the inter-prediction process and the derived extended reference frame, the one or more processors are further configured to:
rotate a current prediction unit in a current cube face of the encoded frame of 360-degree video data based on a rotation angle of a reference cube face containing a reference block for the current prediction unit.

### Claim 13 (depends on 7)

The apparatus of claim 7, further comprising:
a display configured to display at least a portion of the decoded sphere of 360-degree video data.

### Claim 14

An apparatus configured to decode 360-degree video data, the apparatus comprising:
means for receiving an encoded frame of 360-degree video data, the encoded frame of 360-degree video data being arranged in packed faces obtained from a projection of a sphere of the 360-degree video data;
means for decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data being arranged in the packed faces;
means for deriving a decoded sphere of 360-degree video data from the decoded frame of 360-degree video data;
means for sampling the decoded sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the decoded frame of 360-degree video data;
means for deriving an extended reference frame from the extended faces; and
means for decoding a subsequent encoded frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 15

A computer-readable storage medium storing instructions that, when executed, causes one or more processors of a device configured to decode video data to:
receive the encoded frame of 360-degree video data, the encoded frame of 360-degree video data being arranged in packed faces obtained from a projection of a sphere of the 360-degree video data;
decode the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data being arranged in the packed faces;
derive a decoded sphere of 360-degree video data from the decoded frame of 360-degree video data;
sample the decoded sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the decoded frame of 360-degree video data;
derive an extended reference frame from the extended faces; and
decode a subsequent encoded frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 16

A method of encoding 360-degree video data, the method comprising:
receiving a sphere of 360-degree video data;
arranging the sphere of 360-degree video data into a frame of packed faces obtained from a projection of the sphere of 360-degree video data;
encoding the frame of packed faces to form a frame of encoded 360-degree video data;
reconstructing the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data being arranged in the packed faces;
deriving a reconstructed sphere of 360-degree video data from the reconstructed frame of 360-degree video data;
sampling the reconstructed sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the reconstructed frame of 360-degree video data;
deriving an extended reference frame from the extended faces; and
encoding a subsequent frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 17 (depends on 16)

The method of claim 16, wherein the projection is a cubemap projection or an adjusted cubemap projections (ACP).

### Claim 18 (depends on 16)

The method of claim 16, wherein sampling the reconstructed sphere of 360-degree video data to produce extended faces using the projection comprises sampling the reconstructed sphere of 360-degree video data to produce extended faces using the projection in accordance with a number of extension pixels.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
generating an indication of the number of extension pixels at a picture level or a sequence level.

### Claim 20 (depends on 18)

The method of claim 18, wherein the number of extension pixels is equal to a largest prediction unit size in a coding tree unit (CTU) of the frame of packed faces.

### Claim 21 (depends on 16)

The method of claim 16, wherein encoding the subsequent frame of 360-degree video data using the inter-prediction process and the derived extended reference frame comprises:
rotating a current prediction unit in a current cube face of the frame of packed faces based on a rotation angle of a reference cube face containing a reference block for the current prediction unit.

### Claim 22

An apparatus configured to encode 360-degree video data, the apparatus comprising:
a memory configured to store a sphere of 360-degree video data; and
one or more processors in communication with the memory, the one or more processors configured to:
receive the sphere of 360-degree video data;
arrange the sphere of 360-degree video data into a frame of packed faces obtained from a projection of the sphere of 360-degree video data;
encode the frame of packed faces to form a frame of encoded 360-degree video data;
reconstruct the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data being arranged in the packed faces;
derive a reconstructed sphere of 360-degree video data from the reconstructed frame of 360-degree video data;
sample the reconstructed sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the reconstructed frame of 360-degree video data;
derive an extended reference frame from the extended faces; and
encode a subsequent frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the projection is a cubemap projection or an adjusted cubemap projections (ACP).

### Claim 24 (depends on 22)

The apparatus of claim 22, wherein to sample the reconstructed sphere of 360-degree video data to produce extended faces using the projection, the one or more processors are further configured to sample the reconstructed sphere of 360-degree video data to produce extended faces using the projection in accordance with a number of extension pixels.

### Claim 25 (depends on 24)

The apparatus of claim 24, wherein the one or more processors are further configured to:
generate an indication of the number of extension pixels at a picture level or a sequence level.

### Claim 26 (depends on 24)

The apparatus of claim 24, wherein the number of extension pixels is equal to a largest prediction unit size in a coding tree unit (CTU) of the encoded frame of packed faces.

### Claim 27 (depends on 22)

The apparatus of claim 22, wherein to encode the subsequent frame of 360-degree video data using the inter-prediction process and the derived extended reference frame, the one or more processors are further configured to:
rotate a current prediction unit in a current cube face of the encoded frame of packed faces based on a rotation angle of a reference cube face containing a reference block for the current prediction unit.

### Claim 28 (depends on 22)

The apparatus of claim 22, further comprising:
a camera configured to capture the sphere of 360-degree video data.

### Claim 29

An apparatus configured to encode 360-degree video data, the apparatus comprising:
means for receiving a sphere of 360-degree video data;
means for arranging the sphere of 360-degree video data into a frame of packed faces obtained from a projection of the sphere of 360-degree video data;
means for encoding the frame of packed faces to form a frame of encoded 360-degree video data;
means for reconstructing the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data being arranged in the packed faces;
means for deriving a reconstructed sphere of 360-degree video data from the reconstructed frame of 360-degree video data;
means for sampling the reconstructed sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the reconstructed frame of 360-degree video data;
means for deriving an extended reference frame from the extended faces; and
means for encoding a subsequent frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.

### Claim 30

A computer-readable storage medium storing instructions that, when executed, causes one or more processors of a device configured to encode video data to:
receive a sphere of 360-degree video data;
arrange the sphere of 360-degree video data into a frame of packed faces obtained from a projection of the sphere of 360-degree video data;
encode the frame of packed faces to form a frame of encoded 360-degree video data;
reconstruct the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data being arranged in the packed faces;
derive a reconstructed sphere of 360-degree video data from the reconstructed frame of 360-degree video data;
sample the reconstructed sphere of 360-degree video data to produce extended faces using the projection, wherein the extended faces are larger than the packed faces of the reconstructed frame of 360-degree video data;
derive an extended reference frame from the extended faces; and
encode a subsequent frame of 360-degree video data using an inter-prediction process and the derived extended reference frame.