# US 9,519,996 B2 · Virtual view generating method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,519,996 B2
- Document ID: US9519996B2
- Status: Granted
- Kind: B2
- Application number: 14/570,077
- Filed: 2014-12-15
- Published: 2015-06-18
- Granted: 2016-12-13
- Earliest claimed: 2013-12-16
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Cheolkon Jung; Mingchen Han; Zengzeng Sun; Licheng Jiao
- Web page: https://patentgrep.com/patent/US9519996B2

## CPC

- G06T 15/20: Perspective computation
- G06T 7/593: from stereo images
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A virtual view generating method and apparatus are provided that are used to resolve a technical problem existing in the prior art that there are holes in a virtual view, so as to improve quality of the virtual view. The method includes obtaining an original image and an original depth map of the original image; preprocessing the original depth map based on an adaptive filtering algorithm to obtain a processed depth map; generating a basic virtual view based on the original image and the processed depth map; and performing hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view.

## Claims

### Claim 1

A virtual view generating method, comprising:
obtaining an original image and an original depth map of the original image;
preprocessing the original depth map based on an adaptive filtering algorithm to obtain a processed depth map by sequentially performing, separately using a first convolution kernel and a second convolution kernel, adaptive filtering on the original depth map;
generating a basic virtual view based on the original image and the processed depth map; and
performing hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view,
wherein the original depth map is preprocessed by using the first convolution kernel according a formula:
D
(
t
+
1
)
⁡
(
x
,
y
)
=
1
N
(
t
)
⁡
(
x
,
y
)
⁢
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
D
(
t
)
⁡
(
x
+
i
,
y
+
i
)
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
and
wherein D(t+1)(x,y)=max(D(t+1)(x,y),D(t)(x,y)),
N
(
t
)
⁡
(
x
,
y
)
=
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
t indicates the number of iteration time and t=0, 1, 2 . . . , D(i)(x,y) is a depth value that is of a pixel in a depth map and is input at the tth iteration, D(t+1)(x,y) is a depth value that is of the pixel in the depth map and is obtained after the tth iteration, w(x,y) is the first convolution kernel,
w
(
t
)
⁡
(
x
,
y
)
=
g
⁡
(

∇
D
0
(
t
)
⁡
(
x
,
y
)

)
=
1
1
+

∇
D
0
(
t
)
⁡
(
x
,
y
)

,
|∇D0(t)(x,y)|=(Gx(t)(x,y))2+(Gy(t)(x,y))2, Gxt(x,y) is horizontal gradient that is of the pixel in the depth map and is input at the tth iteration, Gy(t)(x,y) is a vertical gradient that is of the pixel in the depth map and is input at the tth iteration, N(x,y) is a normalization factor, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is vertical-axis offset of the pixel.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the original depth map is preprocessed by using the second convolution kernel according to a formula:
D
(
t
+
1
)
⁡
(
x
,
y
)
=
1
N
(
t
)
⁡
(
x
,
y
)
⁢
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
D
(
t
)
⁡
(
x
+
i
,
y
+
i
)
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
and
wherein D(t+1)(x,y)=max(D(t+1)(x,y),D(t) (x,y)),
N
(
t
)
⁡
(
x
,
y
)
=
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
t indicates the number of iteration times and t=0, 1, 2 . . . , D(t)(x,y) is a depth value that is of a pixel in a depth map and is input at the tth iteration, D(t+1)(x,y) is a depth value that is of the pixel in the depth map and is obtained after the tth iteration, w(x,y) is the second convolution kernel,
w
(
t
)
⁡
(
x
,
y
)
=
{
(
1
-

cos
⁡
(
α
(
t
)
⁡
(
x
,
y
)
)

)
×
(
1
-
g
⁡
(

∇
D
0
(
t
)
⁡
(
x
,
y
)

)
)
if
⁢
⁢

∇
D
0
(
t
)
⁡
(
x
,
y
)

>
h
1
else
,
⁢
⁢
α
(
t
)
⁡
(
x
,
y
)
=
tan
-
1
(
G
y
(
t
)
⁡
(
x
,
y
)
G
x
(
t
)
⁡
(
x
,
y
)
)
,
⁢
⁢

∇
D
0
(
t
)
⁡
(
x
,
y
)

=
(
G
x
(
t
)
⁡
(
x
,
y
)
)
2
+
(
G
y
(
t
)
⁡
(
x
,
y
)
)
2
,
Gx(t)(x,y) is a horizontal gradient that is of the pixel in the depth map and is input at the tth iteration, Gy(t)(x,y) is a vertical gradient that is of the pixel in the depth map and is input at the tth iteration, and h is a threshold.

### Claim 3 (depends on 1)

The method according to claim 1, wherein generating the basic virtual view based on the original image and the processed depth map comprises:
calculating a horizontal pixel-level parallax between a pixel in the virtual view and a pixel in the original image according to a depth value of the pixel in the processed depth map; and
generating the basic virtual view according to the pixel in the original image and the horizontal pixel-level parallax.

### Claim 4 (depends on 1)

The method according to claim 1, wherein performing the hierarchical hole filling on the basic virtual view based on the deconvolution algorithm to generate the virtual view comprises:
performing downsarnpling reduction on the basic virtual view to obtain a reduced image;
performing deconvolution expansion on the reduced image to obtain an expanded image; and
performing, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 5 (depends on 1)

The method according to claim 1, wherein performing the hierarchical hole filling on the basic virtual view based on the deconvolution algorithm to generate the virtual view comprises:
performing depth adaptive reduction on the basic virtual view to obtain a reduced image;
performing deconvolution expansion on the reduced image to obtain an expanded image; and
performing, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the depth adaptive reduction is performed on the basic virtual view according to a formula:
R
k
+
1
⁡
[
x
,
y
]
=
{
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
w
⁡
[
i
+
2
,
j
+
2
]
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
=
25
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
⁢
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
<
25
,
wherein
non_zero
⁢
(
x
,
y
)
=
{
0
,
if
⁢
⁢
R
k
⁡
[
x
,
y
]
=
0
1
,
else
,
fk is a depth adaptive weighted value image corresponding to Rk, Rk+1 is an image after reduction, Rk is an image before reduction, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is a vertical-axis offset of the pixel, and nz(Rk[2x,2y]) is the number of nonzero values, in a 5×5 window at a point [2x, 2y], of Rk.

### Claim 7 (depends on 5)

The method according to claim 5, wherein performing the deconvolution expansion on the reduced image to obtain the expanded image comprises:
performing upsampling on the reduced image to obtain an upsampling result; and
performing deconvolution calculation on the upsampling result to obtain the expanded image.

### Claim 8

A virtual view generating apparatus, comprising:
a memory; and
a processor coupled to the memory and configured to:
obtain an original image and an original depth map of the original image;
preprocess the original depth map based on an adaptive filtering algorithm to obtain a processed depth map by sequentially performing, separately using a first convolution kernel and a second convolution kernel, adaptive filtering on the original depth map;
generate a basic virtual view based on the original image and the processed depth map; and
perform hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view by:
performing depth adaptive reduction on the basic virtual view to obtain a reduced image according to the formula:
R
k
+
1
⁡
[
x
,
y
]
=
{
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
w
⁡
[
i
+
2
,
j
+
2
]
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
=
25
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
⁢
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
<
25
,
wherein
non_zero
⁢
(
x
,
y
)
=
{
0
,
if
⁢
⁢
R
k
⁡
[
x
,
y
]
=
0
1
,
else
,
fk is a depth adaptive weighted value image corresponding to Rk, Rk+1 is an image after reduction, Rk is an image before reduction, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is a vertical-axis offset of the pixel, and nz(Rk[2x,2y]) is the number of nonzero values, in a 5×5 window at a point [2x, 2y], of Rk;
performing deconvolution expansion on the reduced image to obtain an expanded image; and
performing, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein the processor is further configured to:
calculate a horizontal pixel-level parallax between a pixel in the virtual view and a pixel in the original image according to a depth value of a pixel in the processed depth map; and
generate the basic virtual view according to the pixel in the original image and the horizontal pixel-level parallax.

### Claim 10 (depends on 8)

The apparatus according to claim 8, wherein the processor is further configured to:
perform downsampling reduction on the basic virtual view to obtain a reduced image;
perform deconvolution expansion on the reduced image to obtain an expanded image; and
perform, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 11 (depends on 10)

The apparatus according to claim 10, wherein the processor is further configured to:
perform upsampling on the reduced image to obtain an upsampling result; and
perform deconvolution calculation on the upsampling result to obtain the expanded image.

### Claim 12 (depends on 8)

The apparatus according to claim 8, wherein the processor is further configured to:
perform depth adaptive reduction on the basic virtual view to obtain a reduced image;
perform deconvolution expansion on the reduced image to obtain an expanded image; and
perform, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 13

A virtual view generating method, comprising:
obtaining an original image and an original depth map of the original image;
preprocessing the original depth map based on an adaptive filtering algorithm to obtain a processed depth map by sequentially performing, separately using a first convolution kernel and a second convolution kernel, adaptive filtering on the original depth map;
generating a basic virtual view based on the original image and the processed depth map; and
performing hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view,
wherein the original depth map is preprocessed by using the second convolution kernel according to a formula:
D
(
t
+
1
)
⁡
(
x
,
y
)
=
1
N
(
t
)
⁡
(
x
,
y
)
⁢
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
D
(
t
)
⁡
(
x
+
i
,
y
+
i
)
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
and
wherein D(t+1)(x,y)=max(D(t+1)(X,y),D(t)(x,y),
N
(
t
)
⁡
(
x
,
y
)
=
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
t indicates the number of iteration times and t=0, 1, 2 . . . , D(t)(x,y) is a depth value that is of a pixel in a depth map and is input at the tth iteration, D(t+1)(x,y) is a depth value that is of the pixel in the depth map and is obtained after the tth iteration, w(x,y) is the second convolution kernel,
w
(
t
)
⁡
(
x
,
y
)
=
{
(
1
-

cos
⁡
(
α
(
t
)
⁡
(
x
,
y
)
)

)
×
(
1
-
g
⁡
(

∇
D
0
(
t
)
⁡
(
x
,
y
)

)
)
if
⁢
⁢

∇
D
0
(
t
)
⁡
(
x
,
y
)

>
h
1
else
,
⁢
⁢
α
(
t
)
⁡
(
x
,
y
)
=
tan
-
1
(
G
y
(
t
)
⁡
(
x
,
y
)
G
x
(
t
)
⁡
(
x
,
y
)
)
,
⁢
⁢

∇
D
0
(
t
)
⁡
(
x
,
y
)

=
(
G
x
(
t
)
⁡
(
x
,
y
)
)
2
+
(
G
y
(
t
)
⁡
(
x
,
y
)
)
2
,
is a horizontal gradient that is of the pixel in the depth map and is input at the tth iteration, Gy(t)(x,y) is a vertical gradient that is of the pixel in the depth map and is input at the tth iteration, and h is a threshold.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the original depth map is preprocessed by using the first convolution kernel according a formula:
D
(
t
+
1
)
⁡
(
x
,
y
)
=
1
N
(
t
)
⁡
(
x
,
y
)
⁢
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
D
(
t
)
⁡
(
x
+
i
,
y
+
i
)
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
and
wherein D(t+1)(x,y)=max(D(t+1)(x,y),D(t)(x,y)),
N
(
t
)
⁡
(
x
,
y
)
=
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
t indicates the number of iteration times and t=0, 1, 2 . . . , D(t)(x,y) is a depth value that is of a pixel in a depth map and is input at the tth iteration, D(t+1)(x,y) is a depth value that is of the pixel in the depth map and is obtained after the tth iteration, w(x,y) is the first convolution kernel,
w
(
t
)
⁡
(
x
,
y
)
=
g
⁡
(

∇
D
0
(
t
)
⁡
(
x
,
y
)

)
=
1
1
+

∇
D
0
(
t
)
⁡
(
x
,
y
)

,
|∇D0(t)|=Gx(t)(x,y))2+(x,y))2+(Gy(t)(x,y))2, Gx(t)(x,y) is a horizontal gradient that is of the pixel in the depth map and is input at the tth iteration, Gy(t)(x,y) is a vertical gradient that is of the pixel in the depth map and is input at the tth iteration, N(x,y) is a normalization factor, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is a vertical-axis offset of the pixel.

### Claim 15 (depends on 13)

The method according to claim 13, wherein generating the basic virtual view based on the original image and the processed depth map comprises:
calculating a horizontal pixel-level parallax between a pixel in the virtual view and a pixel in the original image according to a depth value of the pixel in the processed depth map; and
generating the basic virtual view according to the pixel in the original image and the horizontal pixel-level parallax.

### Claim 16 (depends on 13)

The method according to claim 13, wherein performing the hierarchical hole filling on the basic virtual view based on the deconvolution algorithm to generate the virtual view comprises:
performing downsampling reduction on the basic virtual view to obtain a reduced image;
performing deconvolution expansion on the reduced image to obtain an expanded image; and
performing, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 17 (depends on 13)

The method according to claim 13, wherein performing the hierarchical hole filling on the basic virtual view based on the deconvolution algorithm to generate the virtual view comprises:
performing depth adaptive reduction on the basic virtual view to obtain a reduced image;
performing deconvolution expansion on the reduced image to obtain an expanded image; and
performing, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

### Claim 18 (depends on 17)

The method according to claim 17, wherein the depth adaptive reduction is performed on the basic virtual view according to a formula:
R
k
+
1
⁡
[
x
,
y
]
=
{
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
w
⁡
[
1
+
2
,
j
+
2
]
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
=
25
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
⁢
f
k
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
⁢
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
<
25
,
and wherein
non_zero
⁢
(
x
,
y
)
=
{
0
,
if
⁢
⁢
R
k
⁡
[
x
,
y
]
=
0
1
,
else
,
fk is a depth adaptive weighted value image corresponding to Rk, Rk+1 is an image after reduction, Rk is an image before reduction, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is a vertical-axis offset of the pixel, and nz(Rk[2x 2y]) is the number of nonzero values, in a 5×5 window at a point [2x, 2y], of Rk.

### Claim 19 (depends on 17)

The method according to claim 17, wherein performing the deconvolution expansion on the reduced image to obtain the expanded image comprises:
performing upsampling on the reduced image to obtain an upsampling result; and
performing deconvolution calculation on the upsampling result to obtain the expanded image.