US 9,519,996 B2Granted

Virtual view generating method and apparatus

AssigneeHuawei Technologies Co., Ltd.

InventorsCheolkon Jung, Mingchen Han, Zengzeng Sun, Licheng Jiao

Life of this patent

Estimating the term…

Loading patent…

Abstract

A virtual view generating method and apparatus are provided that are used to resolve a technical problem existing in the prior art that there are holes in a virtual view, so as to improve quality of the virtual view. The method includes obtaining an original image and an original depth map of the original image; preprocessing the original depth map based on an adaptive filtering algorithm to obtain a processed depth map; generating a basic virtual view based on the original image and the processed depth map; and performing hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view.

Claims19

Dependent claims

A virtual view generating method, comprising:

obtaining an original image and an original depth map of the original image;
preprocessing the original depth map based on an adaptive filtering algorithm to obtain a processed depth map by sequentially performing, separately using a first convolution kernel and a second convolution kernel, adaptive filtering on the original depth map;
generating a basic virtual view based on the original image and the processed depth map;
and
performing hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view,
wherein the original depth map is preprocessed by using the first convolution kernel according a formula:
D
(
t
+
1
)
⁡
(
x
,
y
)
=
1
N
(
t
)
⁡
(
x
,
y
)
⁢
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
D
(
t
)
⁡
(
x
+
i
,
y
+
i
)
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
and
wherein D(t+1)(x,y)=max(D(t+1)(x,y),D(t)(x,y)),
N
(
t
)
⁡
(
x
,
y
)
=
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
t indicates the number of iteration time and t=0, 1, 2 . . . , D(i)(x,y) is a depth value that is of a pixel in a depth map and is input at the tth iteration, D(t+1)(x,y) is a depth value that is of the pixel in the depth map and is obtained after the tth iteration, w(x,y) is the first convolution kernel,
w
(
t
)
⁡
(
x
,
y
)
=
g
⁡
(

∇
D
0
(
t
)
⁡
(
x
,
y
)

)
=
1
1
+

∇
D
0
(
t
)
⁡
(
x
,
y
)

,
|∇D0(t)(x,y)|=(Gx(t)(x,y))2+(Gy(t)(x,y))2, Gxt(x,y) is horizontal gradient that is of the pixel in the depth map and is input at the tth iteration, Gy(t)(x,y) is a vertical gradient that is of the pixel in the depth map and is input at the tth iteration, N(x,y) is a normalization factor, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is vertical-axis offset of the pixel.

164 more elements

A virtual view generating apparatus, comprising:

a memory;
and
a processor coupled to the memory and configured to:
obtain an original image and an original depth map of the original image;
preprocess the original depth map based on an adaptive filtering algorithm to obtain a processed depth map by sequentially performing, separately using a first convolution kernel and a second convolution kernel, adaptive filtering on the original depth map;
generate a basic virtual view based on the original image and the processed depth map;
and
perform hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view by:
performing depth adaptive reduction on the basic virtual view to obtain a reduced image according to the formula:
R
k
+
1
⁡
[
x
,
y
]
=
{
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
w
⁡
[
i
+
2
,
j
+
2
]
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
=
25
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
R
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
⁢
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
∑
i
=
-
2
2
⁢
∑
j
=
-
2
2
⁢
f
k
⁡
[
2
⁢
x
+
i
,
2
⁢
y
+
j
]
non_zero
⁢
(
2
⁢
x
+
i
,
2
⁢
y
+
j
)
,
if
⁢
⁢
nz
⁡
(
R
k
⁡
[
2
⁢
x
,
2
⁢
y
]
)
<
25
,
wherein
non_zero
⁢
(
x
,
y
)
=
{
0
,
if
⁢
⁢
R
k
⁡
[
x
,
y
]
=
0
1
,
else
,
fk is a depth adaptive weighted value image corresponding to Rk, Rk+1 is an image after reduction, Rk is an image before reduction, x is a horizontal coordinate of the pixel, y is a vertical coordinate of the pixel, i is a horizontal-axis offset of the pixel, and j is a vertical-axis offset of the pixel, and nz(Rk[2x,2y]) is the number of nonzero values, in a 5×5 window at a point [2x, 2y], of Rk;
performing deconvolution expansion on the reduced image to obtain an expanded image;
and
performing, by using the expanded image, hierarchical hole filling on the reduced image to generate the virtual view.

245 more elements

A virtual view generating method, comprising:

obtaining an original image and an original depth map of the original image;
preprocessing the original depth map based on an adaptive filtering algorithm to obtain a processed depth map by sequentially performing, separately using a first convolution kernel and a second convolution kernel, adaptive filtering on the original depth map;
generating a basic virtual view based on the original image and the processed depth map;
and
performing hierarchical hole filling on the basic virtual view based on a deconvolution algorithm to generate a virtual view,
wherein the original depth map is preprocessed by using the second convolution kernel according to a formula:
D
(
t
+
1
)
⁡
(
x
,
y
)
=
1
N
(
t
)
⁡
(
x
,
y
)
⁢
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
D
(
t
)
⁡
(
x
+
i
,
y
+
i
)
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
and
wherein D(t+1)(x,y)=max(D(t+1)(X,y),D(t)(x,y),
N
(
t
)
⁡
(
x
,
y
)
=
∑
i
=
-
1
1
⁢
∑
j
=
-
1
1
⁢
w
(
t
)
⁡
(
x
+
i
,
y
+
j
)
,
t indicates the number of iteration times and t=0, 1, 2 . . . , D(t)(x,y) is a depth value that is of a pixel in a depth map and is input at the tth iteration, D(t+1)(x,y) is a depth value that is of the pixel in the depth map and is obtained after the tth iteration, w(x,y) is the second convolution kernel,
w
(
t
)
⁡
(
x
,
y
)
=
{
(
1
-

cos
⁡
(
α
(
t
)
⁡
(
x
,
y
)
)

)
×
(
1
-
g
⁡
(

∇
D
0
(
t
)
⁡
(
x
,
y
)

)
)
if
⁢
⁢

∇
D
0
(
t
)
⁡
(
x
,
y
)

>
h
1
else
,
⁢
⁢
α
(
t
)
⁡
(
x
,
y
)
=
tan
-
1
(
G
y
(
t
)
⁡
(
x
,
y
)
G
x
(
t
)
⁡
(
x
,
y
)
)
,
⁢
⁢

∇
D
0
(
t
)
⁡
(
x
,
y
)

=
(
G
x
(
t
)
⁡
(
x
,
y
)
)
2
+
(
G
y
(
t
)
⁡
(
x
,
y
)
)
2
,
is a horizontal gradient that is of the pixel in the depth map and is input at the tth iteration, Gy(t)(x,y) is a vertical gradient that is of the pixel in the depth map and is input at the tth iteration, and h is a threshold.

281 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/570,077Filed 15 Dec 2014
Priority: CN 2013 1 069033816 Dec 2013
Published: US 2015/0170404 A118 Jun 2015
Granted: 13 Dec 2016
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06T 15/20
Geometric effects: perspective computation
G06T 7/593
From multiple images: from stereo images
G06T 19/006
Manipulating three-dimensional [3D] models or images for computer graphics: mixed reality
H04N 13/111
Processing image signals: transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation

Family

US 2015/0170404 A1Pre-grant publication · published 18 Jun 2015
US 9,519,996 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key