# US 10,771,814 B2 · Hybrid video coding supporting intermediate view synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,771,814 B2
- Document ID: US10771814B2
- Status: Granted
- Kind: B2
- Application number: 16/403,887
- Filed: 2019-05-06
- Published: 2019-08-22
- Granted: 2020-09-08
- Earliest claimed: 2010-07-15
- Assignees: Ge Video Compression, LLC
- Applicants: GE Video Compression, LLC
- Inventors: Thomas Wiegand; Karsten Mueller; Philipp Merkle
- Web page: https://patentgrep.com/patent/US10771814B2

## CPC

- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/109: among a plurality of temporal predictive coding modes
- H04N 19/11: among a plurality of spatial predictive coding modes
- H04N 19/124: Quantisation
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/172: the region being a picture, frame or field
- H04N 19/177: the unit being a group of pictures [GOP]
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/513: Processing of motion vectors
- H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors}
- H04N 19/567: Motion estimation based on rate distortion criteria
- H04N 19/61: in combination with predictive coding
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Hybrid video decoder supporting intermediate view synthesis of an intermediate view video from a first- and a second-view video which are predictively coded into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal having a prediction mode is provided, having: an extractor configured to respectively extract, from the multi-view data signal, for sub-regions of the frames of the second-view video, a disparity vector and a prediction residual; a predictive reconstructor configured to reconstruct the sub-regions of the frames of the second-view video, by generating a prediction from a reconstructed version of a portion of frames of the first-view video using the disparity vectors and a prediction residual for the respective sub-regions; and an intermediate view synthesizer configured to reconstruct first portions of the intermediate view video.

## Claims

### Claim 1

A decoder for decoding multi-view data, comprising:
an extractor configured to:
receive multi-view data comprising data related to a first-view video and a second-view video, and
obtain, from the multi-view data, a disparity vector and a prediction residual associated with a sub-region of a frame of the second-view video, wherein the disparity vector indicates a displacement of the sub-region of the frame of the second-view video with respect to a frame of the first-view video;
a predictive reconstructor configured to reconstruct the sub-region of the frame of the second-view video based on a reconstructed portion of the frame of the first-view video, the disparity vector, and the prediction residual, wherein the reconstructed portion of the frame of the first-view video is obtained based on a prediction mode indicated in the data stream; and
a view synthesizer configured to:
determine a scaled disparity vector based on the disparity vector and a scaling factor, and
reconstruct a portion of a frame of a third-view video using the reconstructed portion of the frame of the first view video and the scaled disparity vector.

### Claim 2 (depends on 1)

The decoder of claim 1, wherein the scaling factor is a value between 0 and 1, and the view synthesizer is configured to multiply the disparity vector with the scaling factor to determine the scaled disparity vector.

### Claim 3 (depends on 1)

The decoder of claim 1, wherein the scaling factor is a value less than 0 or greater than 1, and the view synthesizer is configured to multiply the disparity vector with the scaling factor to determine the scaled disparity vector.

### Claim 4 (depends on 1)

The decoder of claim 1, wherein the scaling factor depends on a spatial location of a third view corresponding to the third-view video relative to a first view corresponding to the first-view video and a second view corresponding to the second-view video.

### Claim 5 (depends on 1)

The decoder of claim 1, wherein the scaled disparity vector and the disparity vector are equal.

### Claim 6 (depends on 1)

The decoder of claim 1, wherein the view synthesizer is further configured to extrapolate or interpolate another portion of the frame of the third-view video based on the reconstructed portion of the frame of the third-view video.

### Claim 7 (depends on 1)

The decoder of claim 1, wherein the view synthesizer is further configured to: determine an interpolated disparity vector based on a plurality of disparity vectors associated with different sub-regions of the second-view video, and reconstruct another portion of the frame of the third-view video using the reconstructed portion of the frame of the first view video and the interpolated disparity vector.

### Claim 8 (depends on 7)

The decoder of claim 7, wherein the different sub-regions of the second-view video belong to a same frame of the second-view video.

### Claim 9 (depends on 7)

The decoder of claim 7, wherein the different sub-regions of the second-view video belong to different frames of the second-view video.

### Claim 10 (depends on 1)

The decoder of claim 1, wherein the extractor is further configured to extract, from the multi-view data, reliability data indicating a reliability factor for the disparity vector, wherein the view synthesizer is configured to reconstruct the portion of the frame of the third-view video based on the reliability data satisfying a predetermined requirement.

### Claim 11

An encoder for encoding multi-view data, comprising:
a prediction unit configured to encode multi-view data comprising data related to a first-view video and a second-view video, the data including a prediction mode used to encode the first-view video,
wherein to encode the multi-view data, the prediction unit is at least configured to determine a disparity vector and a prediction residual associated with a sub-region of a frame of the second-view video, the disparity vector indicating a displacement of the sub-region of the frame of the second-view video with respect to a frame of the first-view video; and
a data signal generator configured to insert the multi-view data including the disparity vector and the prediction residual into a data stream, wherein data associated with a portion of the frame of the first-view video and a scaled disparity vector are used to synthesize a portion of a frame of a third-view video, and the scaled disparity vector is determined using the disparity vector and a scaling factor.

### Claim 12 (depends on 11)

The encoder of claim 11, wherein the scaling factor is a value between 0 and 1, and the scaled disparity vector is determined by multiplying the disparity vector with the scaling factor.

### Claim 13 (depends on 11)

The encoder of claim 11, wherein the scaling factor is a value less than 0 or greater than 1, and the scaled disparity vector is determined by multiplying the disparity vector with the scaling factor.

### Claim 14 (depends on 11)

The encoder of claim 11, wherein the scaling factor depends on a spatial location of a third view corresponding to the third-view video relative to a first view corresponding to the first-view video and a second view corresponding to the second-view video.

### Claim 15 (depends on 11)

The encoder of claim 11, wherein the synthesized portion of the frame of the third view video is used to extrapolate or interpolate another portion of the frame of the third-view video.

### Claim 16 (depends on 11)

The encoder of claim 11, wherein an interpolated disparity vector is determined based on a plurality of disparity vectors associated with different sub-regions of the second-view video, and another portion of the frame of the third-view video is synthesized using the synthesized portion of the frame of the first view video and the interpolated disparity vector.

### Claim 17 (depends on 16)

The encoder of claim 16, wherein the different sub-regions of the second-view video belong to a same frame of the second-view video.

### Claim 18 (depends on 16)

The encoder of claim 16, wherein the different sub-regions of the second-view video belong to different frames of the second-view video.

### Claim 19 (depends on 11)

The encoder of claim 11, wherein the prediction unit is further configured to encode, as part of the multi-view data, reliability data indicating a reliability factor for the disparity vector, wherein the portion of the frame of the third-view video is synthesized based on the reliability data satisfying a predetermined requirement.

### Claim 20

A non-transitory computer-readable medium for storing data associated with a video, comprising:
a data stream stored in the non-transitory computer-readable medium, the data stream comprising multi-view data including at least a first-view video and a second-view video, the multi-view data comprising a prediction mode used to encode the first-view video, and a disparity vector and a prediction residual associated with a sub-region of a frame of the second-view video, wherein the disparity vector indicates a spatial displacement of the sub-region of the frame of the second-view video with respect to a frame of the first-view video,
wherein data associated with a portion of the frame of the first-view video and a scaled disparity vector are used to synthesize a portion of a frame of a third-view video, and the scaled disparity vector is determined using the disparity vector and a scaling factor.