# US 2023/0419519 A1 · Depth estimation method in an immersive video context

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419519 A1
- Document ID: US20230419519A1
- Status: Published application
- Kind: A1
- Application number: 18/367,337
- Filed: 2023-09-12
- Published: 2023-12-28
- Earliest claimed: 2021-10-05
- Assignees: Tencent Cloud Europe (France) SAS
- Applicants: Tencent Cloud Europe (France) SAS
- Inventors: Joël JUNG
- Web page: https://patentgrep.com/patent/US20230419519A1

## CPC

- G06T 7/50: Depth or shape recovery
- G06T 7/55: from multiple images
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/172: image signals comprising non-image signal components, e.g. headers or format information
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of synthesizing an image includes receiving a coded video stream corresponding to different views of a scene acquired by a plurality of cameras, and identifying one or more derivable depth blocks of a target depth map corresponding to an image of the coded video stream. The method further includes determining a corresponding operation to derive a depth of each of the one or more derivable depth blocks, the operation being applied to a depth of a reference depth block included in a depth map of a reference image in the coded video bitstream. The method further includes obtaining the target depth map corresponding to the image by executing the corresponding operation of the respective derivable depth block, and synthesizing a new image according to the target depth map and at least one decoded image from the coded video stream.

## Claims

### Claim 1

A method of synthesizing an image, the method comprising:
receiving, from a server, a coded video stream corresponding to different views of a scene acquired by a plurality of cameras;
identifying one or more derivable depth blocks of a target depth map corresponding to an image of the coded video stream;
determining, for each of the one or more derivable depth blocks, a corresponding operation to derive a depth of the respective one of the one or more derivable depth blocks, the operation being applied to a depth of at least one reference depth block included in a depth map of a reference image in the coded video bitstream;
obtaining the target depth map corresponding to the image by executing, for each of the one or more derivable depth blocks, the corresponding operation of the respective derivable depth block; and
synthesizing a new image according to the target depth map and at least one decoded image from the coded video stream.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the identifying is based on a signal received from the server.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the signal is received in a supplemental enhancement information message.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising:
decoding the coded video stream to obtain decoded images; and
identifying the one or more derivable depth blocks of the target depth map corresponding to a decoded image of the decoded images, wherein each of the identified one or more derivable depth blocks has a depth that is derivable by executing an operation on a depth of at least one reference depth block included in a depth map of a reference image among the decoded images.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the one or more derivable depth blocks are obtained by parsing the coded video stream.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the one or more derivable depth blocks are determined by analyzing the decoded images.

### Claim 7 (depends on 4)

The method according to claim 4, further comprising obtaining the at least one reference depth block by at least one of:
using a motion estimation method applied to at least one of the decoded images; or
using a motion estimation method applied to at least one depth map corresponding to at least one of the decoded images; or
generating a list of at least one candidate block determined for a respective derivable depth block, wherein the at least one candidate block is:
a block adjacent to the respective derivable depth block in the target depth map of the image; or
a block at a location of the respective derivable depth block in a depth map of another decoded image of the decoded images.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the signal further indicates a respective operation for deriving each of the derivable depth blocks.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the signal further comprises a respective identifier of the at least one reference depth block for deriving each of the derivable depth locks.

### Claim 10 (depends on 1)

The method according to claim 1, wherein a respective operation for deriving the depth of each of the one or more derivable depth blocks does not use intrinsic or extrinsic parameters of the plurality of cameras.

### Claim 11 (depends on 1)

The method according to claim 1, wherein a respective operation for deriving the depth of one of the one or more derivable depth blocks is a linear combination of depths of two or more reference depth blocks.

### Claim 12

A method comprising:
obtaining a plurality of images corresponding to different views of a scene acquired by a plurality of cameras;
generating a coded video stream by encoding the images;
identifying one or more derivable depth blocks of a target depth map corresponding to an image of the plurality of images, wherein each of the identified one or more derivable depth blocks has a depth that is derivable by executing an operation on a depth of at least one reference depth block included in a depth map of a reference image among the plurality of images;
generating a signal indicating the one or more derivable depth blocks;
sending the coded video stream and the signal to a decoder device according to a lossless protocol.

### Claim 13 (depends on 12)

The method according to claim 12, further comprising obtaining the at least one reference depth block by at least one of:
using a motion estimation method applied to at least one of the plurality of images; or
using a motion estimation method applied to at least one depth map corresponding to at least one of the plurality of images; or
generating a list of at least one candidate block determined for a respective derivable depth block, wherein the at least one candidate block is:
a block adjacent to the respective derivable depth block in the target depth map corresponding to the image; or
a block at a location of the respective derivable depth block in a depth map of another image of the plurality of images.

### Claim 14 (depends on 12)

The method according to claim 12, comprising:
obtaining a viewpoint selected by an observer;
determining at least one unnecessary depth block of the target depth map, wherein a depth of the at least one unnecessary depth block is not necessary for synthesizing a new image according to the target depth map and at least one of the plurality of images; and
indicating, in the signal, the at least one unnecessary depth block.

### Claim 15 (depends on 12)

The method according to claim 12, wherein the signal further indicates a respective operation for deriving each of the derivable depth blocks.

### Claim 16 (depends on 12)

The method according to claim 12, wherein the signal further comprises a respective identifier of the at least one reference depth block F for deriving each of the derivable depth locks.

### Claim 17 (depends on 12)

The method according to claim 12, wherein a respective operation for deriving the depth of one of the one or more derivable depth blocks does not use intrinsic or extrinsic parameters of the plurality of cameras.

### Claim 18 (depends on 12)

The method according to claim 12, wherein a respective operation for deriving the depth of one of the one or more derivable depth blocks is a linear combination of depths of two or more reference depth blocks.

### Claim 19 (depends on 12)

The method according to claim 12, wherein the respective operation for deriving the depth of one of the one or more derivable depth blocks is a copy of a depth of a reference depth block.

### Claim 20

An apparatus, comprising:
processing circuitry configured to
receive, from a server, a coded video stream corresponding to different views of a scene acquired by a plurality of cameras;
identify one or more derivable depth blocks of a target depth map corresponding to an image of the coded video stream;
determine, for each of the one or more derivable depth blocks, a corresponding operation to derive a depth of the respective one of the one or more derivable depth blocks, the operation being applied to a depth of at least one reference depth block included in a depth map of a reference image in the coded video bitstream;
obtain the target depth map corresponding to the image by executing, for each of the one or more derivable depth blocks, the corresponding operation of the respective derivable depth block; and
synthesize a new image according to the target depth map and at least one decoded image from the coded video stream.
