# US 12,183,020 B2 · Method and apparatus to complement depth image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,020 B2
- Document ID: US12183020B2
- Status: Granted
- Kind: B2
- Application number: 17/411,200
- Filed: 2021-08-25
- Published: 2022-03-03
- Granted: 2024-12-31
- Earliest claimed: 2020-08-31
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Zhaohui Lv; Weijia Zhang; Mingming Fan
- Web page: https://patentgrep.com/patent/US12183020B2

## CPC

- G06T 7/55: from multiple images
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06V 10/56: relating to colour
- G06V 10/82: using neural networks
- G06V 30/1918: {Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion}
- G06N 3/048: Activation functions
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus for complementing a depth image are provided. The method includes obtaining a color image and a corresponding depth image, obtaining a first depth image based on the color image, using a first deep neural network (DNN), obtaining a second depth image based on the depth image and an intermediate feature image generated by each intermediate layer of the first DNN, using a second DNN, and obtaining a final depth image by merging the first depth image and the second depth image.

## Claims

### Claim 1

A processor-implemented method of complementing a depth image, comprising:
obtaining a first depth image based on only a color image, using a first deep neural network (DNN);
obtaining a second depth image based on only an original depth image, corresponding to the color image, and an intermediate feature image generated by each intermediate layer of the first DNN, using a second DNN to perform feature fusion including inputting the intermediate feature image from each intermediate layer of the first DNN to each corresponding intermediate layer of the second DNN; and
obtaining a final depth image by merging the first depth image and the second depth image,
wherein the feature fusion is only performed at the second DNN.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the first DNN comprises a first encoder network and a having an N-layer residual structure,
wherein the second DNN comprises a second encoder network and a second decoder network having the N-layer residual structure,
wherein N is an integer greater than 1, and
wherein the obtaining of the second depth image further comprises performing feature decoding using the second decoder network, based on an output of the first encoder network, an output of the second encoder network, an intermediate feature image of the first decoder network, and an intermediate feature image of the second encoder network.

### Claim 3 (depends on 2)

The method of claim 2, wherein the obtaining of the second depth image further comprises:
performing feature encoding using the second encoder network of the second DNN, based on the color image and an intermediate feature image of the first encoder network.

### Claim 4 (depends on 2)

The method of claim 2,
wherein the first DNN further comprises a first preprocessing network preceding the first encoder network, and a first depth prediction network after the first decoder network,
wherein the obtaining of the first depth image comprises:
transforming the color image into a first feature image for DNN processing using the first preprocessing network and inputting the first feature image to the first encoder network; and
synthesizing, into the first depth image, a feature image output from the first decoder network using the first depth prediction network,
wherein the second DNN further comprises a second preprocessing network preceding the second encoder network, and a second depth prediction network after the second decoder network, and
wherein the obtaining of the second depth image comprises:
transforming the original depth image into a second feature image for DNN processing using the second preprocessing network and inputting the second feature image to the second encoder network; and
obtaining the second depth image by fusing the second feature image, the feature image output by the first decoder network, and a feature image output by the second decoder network, using the second depth prediction network.

### Claim 5 (depends on 4)

The method of claim 4,
wherein an input of a first layer of the second decoder network is a sum of a feature image output from the second encoder network and a feature image output from the first encoder network,
wherein an input of each layer of the second decoder network, from a second layer up to an Nth layer of the second decoding network, comprises a respective feature image obtained by fusing a feature image output from a previous layer of the second decoder network, a corresponding feature image output from a corresponding layer of the first decoder network, and another corresponding feature image output from a corresponding layer of the second encoder network, using a squeeze-and-excitation (SE) block-based method, and
wherein an input of the second depth prediction network comprises a feature image obtained by fusing the feature image output from the second decoder network, the feature image output from the first decoder network, and the second feature image using the SE block-based method.

### Claim 6 (depends on 5)

The method of claim 5, wherein an input of a first layer of the second encoder network comprises a sum of the first feature image and the second feature image, and
an input of each layer of the second encoder network, from a second layer up to an Nth layer of the second encoder network, is a sum of a feature image output from a previous layer of the second encoder network and a feature image output from a corresponding layer of the first encoder network.

### Claim 7 (depends on 2)

The method of claim 2, wherein a respective residual block of each of the second encoder network and the second decoder network is configured to execute a gated process in response to executing a convolution process.

### Claim 8 (depends on 1)

The method of claim 1, wherein the obtaining of the final depth image by merging the first depth image and the second depth image comprises:
obtaining a first pixel weight map of the first depth image and a second pixel weight map of the second depth image, using an attention network; and
obtaining the final depth image by assigning a respective weight to each of the first depth image and the second depth image based on the first pixel weight map and the second pixel weight map and obtaining a sum therefrom.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
generating a loss function based on a first mean squared error (MSE) loss of a training first depth image and an actual depth image, a second MSE loss of a training second depth image and the actual depth image, a third MSE loss of a training final depth image and the actual depth image, and a structural loss of the training final depth image and the actual depth image; and
generate the first DNN, the second DNN, and/or the attention network by respectively training an in-training first DNN, an in-training second DNN, and/or an in-training attention network using the loss function before the use of the first DNN, the use of the second DNN, and/or the use of the attention network,
wherein the structural loss comprises a structural similarity index measure (SSIM),
wherein the training first depth image is obtained using the in-training first DNN based on a training color image,
wherein the training second depth image is obtained using the in-training second DNN based on a training depth image corresponding to the training color image, and a training intermediate feature image generated by each intermediate layer of the in-training first DNN, and
wherein the training final depth image is obtained by merging the training first depth image and the training second depth image.

### Claim 10 (depends on 9)

The method of claim 9, wherein the generating of the loss function further comprises obtaining the loss function based on a weighted summation of the first MSE loss, the second MSE loss, the third MSE loss, and the structural loss.

### Claim 11 (depends on 1)

The method of claim 1, wherein the obtaining of the color image and the original depth image further comprises:
obtaining, as the depth image, a depth image of which a pixel value is 0, in response to an absence of the depth image.

### Claim 12 (depends on 1)

The method of claim 1, wherein the feature fusion is performed at the second DNN by combining the intermediate feature image of the first DNN with the corresponding intermediate layer of the second DNN.

### Claim 13 (depends on 1)

The method of claim 1, wherein each of the intermediate layers of the first DNN is a single layer of a respective block, of plural layers, of the first DNN.

### Claim 14 (depends on 13)

The method of claim 13, wherein each of the corresponding intermediate layers is single layer of a respective block, of multiple layers, of the second DNN.

### Claim 15 (depends on 14)

The method of claim 14, wherein at least one of the respective blocks of the first DNN comprises two convolution layers and/or at least one of the respective blocks of the second DNN comprises two convolution layers.

### Claim 16 (depends on 1)

A non-transitory computer-readable storage medium storing commands that, when executed by a processor, cause the processor to perform the method of claim 1.

### Claim 17

An apparatus for complementing a depth image, comprising:
one or more processors configured to:
obtain a first depth image based on only a color image, using a first deep neural network (DNN);
obtain a second depth image based on only an original depth image, corresponding to the color image, and an intermediate feature image generated by each intermediate layer of the first DNN, using a second DNN to perform feature fusion including inputting the intermediate feature image from each intermediate layer of the first DNN to each corresponding intermediate layer of the second DNN; and
obtain a final depth image by merging the first depth image and the second depth image,
wherein the feature fusion is only performed at the second DNN.

### Claim 18 (depends on 17)

The apparatus of claim 17, further comprising:
a memory storing instructions that when executed by the one or more processors configures the one or more processors to perform the obtaining of the first depth image, the obtaining of the second depth image, and the obtaining of the final depth image.

### Claim 19 (depends on 17)

The apparatus of claim 17,
wherein the first DNN comprises a first encoder network and a first decoder network having an N-layer residual structure,
wherein the second DNN comprises a second encoder network and a second decoder network having an N-layer residual structure,
wherein N is an integer greater than 1, and
wherein, for the obtaining of the second depth image, the one or more processors are further configured to perform feature decoding using the second decoder network, based on an output of the first encoder network, an output of the second encoder network, an intermediate feature image of the first decoder network, and an intermediate feature image of the second encoder network.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein, for the obtaining of the second depth image, the one or more processors are configured to perform feature encoding using the second encoder network of the second DNN, based on the original depth image and an intermediate feature image of the first encoder network.

### Claim 21 (depends on 19)

The apparatus of claim 19,
wherein the first DNN further comprises a first preprocessing network preceding the first encoder network, and a first depth prediction network after the first decoder network,
wherein, for the obtaining of the first depth image, the one or more processors are further configured to:
transform the color image into a first feature image for DNN processing using the first preprocessing network, input the first feature image to the first encoder network; and
synthesize, into the first depth image, a feature image output from the first decoder network using the first depth prediction network,
wherein the second DNN further comprises a second preprocessing network preceding the second encoder network and a second depth prediction network after the second decoder network, and
wherein, for the he obtaining of the second depth image, the one or more processors are further configured to:
transform the original depth image into a second feature image for DNN processing using the second preprocessing network, input the second feature image to the second encoder network; and
obtain the second depth image by fusing the second feature image, the feature image output by the first decoder network, and the feature image output by the second decoder network, using the second depth prediction network.

### Claim 22 (depends on 17)

The apparatus of claim 17, wherein, for the obtaining of the final depth image, the one or more processors are further configured to:
obtain a first pixel weight map of the first depth image and a second pixel weight map of the second depth image, using an attention network; and
obtain the final depth image by assigning a respective weight to each of the first depth image and the second depth image based on the first pixel weight map and the second pixel weight map and obtaining a sum therefrom.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the one or more processors are further configured to:
generate a loss function based on a first mean squared error (MSE) loss of a training first depth image and an actual depth image, a second MSE loss of a training second depth image and the actual depth image, a third MSE loss of a training final depth image and the actual depth image, and a structural loss of the training final depth image and the actual depth image; and
generate the first DNN, the second DNN, and/or the attention network by respectively training an in-training first DNN, an in-training second DNN, and/or an in-training attention network using the loss function before the use of the first DNN, the use of the second DNN, and/or the use of the attention network,
wherein the structural loss comprises a structural similarity index measure (SSIM),
wherein the training first depth image is obtained using the in-training first DNN based on a training color image, and
wherein the training second depth image is obtained using the in-training second DNN based on a training depth image corresponding to the training color image, and a training intermediate feature image generated by each intermediate layer of the in-training first DNN, and
wherein the training final depth image is obtained by merging the training first depth image and the training second depth image.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein, for the generation of the loss function, the one or more processors are configured to:
generate the loss function based on a weighted summation of the first MSE loss, the second MSE loss, the third MSE loss, and the structural loss.

### Claim 25 (depends on 14)

The apparatus of claim 14, wherein each of the intermediate layers of the first DNN is a single layer of a respective block, of plural layers, of the first DNN.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein at least one of the respective blocks of the first DNN comprises two convolution layers.

### Claim 27 (depends on 26)

The apparatus of claim 26, wherein each of the corresponding intermediate layers is single layer of a respective block, of multiple layers, of the second DNN, and wherein at least one of the respective blocks of the second DNN comprises two convolution layers.