# US 12,008,776 B2 · Depth map processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,776 B2
- Document ID: US12008776B2
- Status: Granted
- Kind: B2
- Application number: 17/436,698
- Filed: 2020-02-13
- Published: 2022-05-19
- Granted: 2024-06-11
- Earliest claimed: 2019-03-08
- Assignees: InterDigital VC Holdings, Inc.
- Applicants: InterDigital VC Holdings, Inc.
- Inventors: Didier Doyen; Benoit Vandame; Guillaume Boisson
- Web page: https://patentgrep.com/patent/US12008776B2

## CPC

- G06T 7/50: Depth or shape recovery
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- G06T 2207/10016: Video; Image sequence
- H04N 13/128: Adjusting depth or disparity
- H04N 13/194: Transmission of image signals
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 2213/003: Aspects relating to the "2D+depth" image format
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

For multi-view video content represented in the MVD (Multi-view+Depth) format, the depth maps may be processed to improve the coherency therebetween. In one implementation, to process a target view based on an input view, pixels of the input view are first projected into the world coordinate system, then into the target view to form a projected view. The texture of the projected view and the texture of the target view are compared. If the difference at a pixel is small, then the depth of the target view at that pixel is adjusted, for example, replaced by the corresponding depth of the projected view. When the multi-view video content is encoded and decoded in a system, depth map processing may be applied in the pre-processing and post-processing modules to improve video compression efficiency and the rendering quality.

## Claims

### Claim 1

A method of video processing, comprising:
accessing a first view and a second view of multi-view video content;
projecting a first pixel at a first position in the first view onto a second position in the second view;
determining a difference between texture information of the first pixel of the first view and texture information of a second pixel at the second position in the second view by evaluating photo-consistency between the first pixel of the first view and the second pixel of the second view; and
adjusting a depth value of the second pixel of the second view based on a depth value of the first pixel of the first view, based on a comparison of the difference with a threshold value.

### Claim 2 (depends on 1)

The method of claim 1, wherein the projecting comprises:
projecting the first pixel into a fourth position in a world coordinate system, based on pose and inverse intrinsic parameters associated with a camera used to capture the first view;
projecting the fourth position in the world coordinate system into a fifth position in the second view based on intrinsic and extrinsic parameters associated with a camera used to capture the second view; and
obtaining the second position from the fifth position.

### Claim 3 (depends on 2)

The method of claim 2, wherein the obtaining comprises selecting a nearest pixel position of the fifth position as the second position.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first view is a neighbor view of the second view.

### Claim 5 (depends on 1)

The method of claim 1, wherein the difference is determined based on respective differences of three components of the texture information of the first pixel of the first view and the texture information of the second pixel of the second view.

### Claim 6 (depends on 1)

The method of claim 1, wherein the difference is determined based on respective differences of three components of texture information of a group of pixels around the first pixel of the first view and texture information of a corresponding group of pixels around the second pixel of the second view.

### Claim 7 (depends on 1)

The method of claim 1, wherein the adjusting is performed if the difference is smaller than the threshold value.

### Claim 8 (depends on 1)

The method of claim 1, wherein the adjusting further comprises applying pose and intrinsic parameters associated with the second view.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
projecting the second pixel of the second view onto a third position in the first view; and
determining a second difference between the texture information of the second pixel of the second view and texture information of a third pixel at the third position in the first view, wherein the adjustment is performed only if the second difference is smaller than a second value.

### Claim 10 (depends on 1)

The method of claim 1, wherein the projecting is performed for all pixels in the first view.

### Claim 11 (depends on 10)

The method of claim 10, wherein the projection result is filtered.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first view is represented by a first texture image and a first depth map, and wherein the second view is represented by a second texture image and a second depth map.

### Claim 13 (depends on 1)

The method of claim 1, wherein the multi-view video content is represented in a Multi-view plus Depth (MVD) format.

### Claim 14

An apparatus for video processing, comprising one or more processors, wherein the one or more processors are configured to:
access a first view and a second view of multi-view video content;
project a first pixel at a first position in the first view onto a second position in the second view;
determine a difference between texture information of the first pixel of the first view and texture information of a second pixel at the second position in the second view by evaluating photo-consistency between the first pixel of the first view and the second pixel of the second view; and
adjust a depth value of the second pixel of the second view based on a depth value of the first pixel of the first view, based on a comparison of the difference with a threshold value.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the one or more processors are configured to:
project the first pixel into a fourth position in a world coordinate system, based on pose and inverse intrinsic parameters associated with a camera used to capture the first view;
project the fourth position in the world coordinate system into a fifth position in the second view based on intrinsic and extrinsic parameters associated with a camera used to capture the second view; and
obtain the second position from the fifth position.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the one or more processors are configured to select a nearest pixel position of the fifth position as the second position.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the first view is a neighbor view of the second view.

### Claim 18 (depends on 14)

The apparatus of claim 14, wherein the difference is determined based on respective differences of three components of the texture information of the first pixel of the first view and the texture information of the second pixel of the second view.

### Claim 19 (depends on 14)

The apparatus of claim 14, wherein the difference is determined based on respective differences of three components of texture information of a group of pixels around the first pixel of the first view and texture information of a corresponding group of pixels around the second pixel of the second view.

### Claim 20 (depends on 14)

The apparatus of claim 14, wherein the adjustment is performed if the difference is smaller than the threshold value.

### Claim 21 (depends on 14)

The apparatus of claim 14, wherein the adjustment further comprises applying pose and intrinsic parameters associated with the second view.

### Claim 22 (depends on 14)

The apparatus of claim 14, wherein the one or more processors are further configured to:
project the second pixel of the second view onto a third position in the first view; and
determine a second difference between the texture information of the second pixel of the second view and texture information of a third pixel at the third position in the first view, wherein the adjustment is performed only if the second difference is smaller than a second threshold value.

### Claim 23 (depends on 14)

The apparatus of claim 14, wherein the projection is performed for all pixels in the first view.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein the projection result is filtered.

### Claim 25 (depends on 14)

The apparatus of claim 14, wherein the first view is represented by a first texture image and a first depth map, and wherein the second view is represented by a second texture image and a second depth map.

### Claim 26 (depends on 14)

The apparatus of claim 14, wherein the multi-view video content is represented in a Multi-view plus Depth (MVD) format.

### Claim 27

A non-transitory machine readable medium having stored thereon machine executable instructions that, when executed, implement a method, the method comprising:
accessing a first view and a second view of multi-view video content;
projecting a first pixel at a first position in the first view onto a second position in the second view;
determining a difference between texture information of the first pixel of the first view and texture information of a second pixel at the second position in the second view by evaluating photo-consistency between the first pixel of the first view and the second pixel of the second view; and
adjusting a depth value of the second pixel of the second view based on a depth value of the first pixel of the first view, based on a comparison of the difference with a threshold value.