US 9,869,863 B2Granted

Systems and methods for processing a 2D video

AssigneeUnity IPR ApS

InventorChristophe Souchard

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system includes one or more hardware processors, a head mounted display (HMD), a digital camera device, and a path extraction engine. The path extraction engine is configured to receive two-dimensional (2D) video from the digital camera device, the 2D video presenting a 2D view of a three-dimensional (3D) object, the digital camera device moving along a camera path as the digital camera device captures the 2D video, identify a plurality of 2D features associated with the 3D object within the 2D video, track the plurality of 2D features across multiple frames of the 2D video, estimate motion in pixels associated with each 2D feature of the plurality of 2D features, using the estimated motion in the pixels, identify 2D motion caused by the digital camera device moving along the camera path, and compute a first camera position of the digital camera device based on the identified 2D motion.

Claims14

Dependent claims

A system comprising:

one or more hardware processors;
a head mounted display (HMD) configured to display a virtual environment to a user wearing the HMD;
a digital camera device configured to capture digital video;
and
a path extraction engine, executable by the one or more hardware processors, configured to perform operations comprising:
receiving two-dimensional (2D) video from the digital camera device, the 2D video presenting a 2D view of a three-dimensional (3D) object, the digital camera device moving along a camera path as the digital camera device captures the 2D video;
identifying a plurality of 2D features associated with the 3D object within the 2D video;
tracking the plurality of 2D features across multiple frames of the 2D video;
estimating motion in pixels associated with each 2D feature of the plurality of 2D features;
using the estimated motion in the pixels, identifying a component of 2D motion caused by the digital camera device moving along the camera path;
and
computing a first camera position of the digital camera device along the camera path based on the identified component of 2D motion,
wherein identifying the plurality of 2D features further includes:
computing a score for a first pixel of a plurality of pixels in a first frame based on a neighborhood of pixels around the first pixel;
and
selecting the first pixel as a 2D feature of the plurality of features based on the score for the first pixel exceeding a pre-determined threshold,
wherein computing the score further includes computing a second moment matrix V on the neighborhood of pixels U around each pixel as:
V
=
[
∑
U
⁢
(
∂
I
∂
x
)
2
∑
U
⁢
(
∂
I
∂
x
)
⁢
(
∂
I
∂
y
)
∑
U
⁢
(
∂
I
∂
x
)
⁢
(
∂
I
∂
y
)
∑
U
⁢
(
∂
I
∂
y
)
2
]
,
where I(x,y,t) is the frame intensity along x and y directions for a given frame t, and
wherein computing the score for the first pixel, ScoreFirstPixel, includes computing:
Score
Pixel
=
det
⁡
(
V
)
Trace
⁡
(
V
)
+
ɛ
,
where ε is a small number used to avoid a division by zero error, det(V) is the determinant of V, and Trace(V) is the trace of V.

90 more elements

A computer-implemented method comprising:

receiving two-dimensional (2D) video from a digital camera device, the 2D video presenting a 2D view of a three-dimensional (3D) object, the digital camera device moving along a camera path as the digital camera device captures the 2D video;
identifying a plurality of 2D features associated with the 3D object within the 2D video;
tracking the plurality of 2D features across multiple frames of the 2D video;
estimating motion in pixels associated with each 2D feature of the plurality of 2D features;
using the estimated motion in the pixels, identifying a component of 2D motion caused by the digital camera device moving along the camera path;
and
computing a first camera position of the digital camera device along the camera path based on the identified component of 2D motion,
wherein identifying the plurality of 2D features further includes:
computing a score for a first pixel of a plurality of pixels in a first frame based on a neighborhood of pixels around the first pixel;
and
selecting the first pixel as a 2D feature of the plurality of features based on the score for the first pixel exceeding a pre-determined threshold,
wherein computing the score further includes computing a second moment matrix V on the neighborhood of pixels U around each pixel as:
V
=
[
∑
U
⁢
(
∂
I
∂
x
)
2
∑
U
⁢
(
∂
I
∂
x
)
⁢
(
∂
I
∂
y
)
∑
U
⁢
(
∂
I
∂
x
)
⁢
(
∂
I
∂
y
)
∑
U
⁢
(
∂
I
∂
y
)
2
]
,
where I(x,y,t) is the frame intensity along x and y directions for a given frame t, and
wherein computing the score for the first pixel, ScoreFirstPixel, includes computing:
Score
Pixel
=
det
⁡
(
V
)
Trace
⁡
(
V
)
+
ɛ
,
where ε is a small number used to avoid a division by zero error, det(V) is the determinant of V, and Trace(V) is the trace of V.

85 more elements

A non-transitory machine-readable medium storing processor-executable instructions which, when executed by a processor, cause the processor to:

receive two-dimensional (2D) video from a digital camera device, the 2D video presenting a 2D view of a three-dimensional (3D) object, the digital camera device moving along a camera path as the digital camera device captures the 2D video;
identify a plurality of 2D features associated with the 3D object within the 2D video;
track the plurality of 2D features across multiple frames of the 2D video;
estimate motion in pixels associated with each 2D feature of the plurality of 2D features;
using the estimated motion in the pixels, identify a component of 2D motion caused by the digital camera device moving along the camera path;
and
compute a first camera position of the digital camera device along the camera path based on the identified component of 2D motion,
wherein identifying the plurality of 2D features further includes:
computing a score for a first pixel of a plurality of pixels in a first frame based on a neighborhood of pixels around the first pixel;
and
selecting the first pixel as a 2D feature of the plurality of features based on the score for the first pixel exceeding a pre-determined threshold,
wherein computing the score further includes computing a second moment matrix V on the neighborhood of pixels U around each pixel as:
V
=
[
∑
U
⁢
(
∂
I
∂
x
)
2
∑
U
⁢
(
∂
I
∂
x
)
⁢
(
∂
I
∂
y
)
∑
U
⁢
(
∂
I
∂
x
)
⁢
(
∂
I
∂
y
)
∑
U
⁢
(
∂
I
∂
y
)
2
]
,
where I(x,y,t) is the frame intensity along x and y directions for a given frame t, and
wherein computing the score for the first pixel, ScoreFirstPixel, includes computing:
Score
Pixel
=
det
⁡
(
V
)
Trace
⁡
(
V
)
+
ɛ
,
where ε is a small number used to avoid a division by zero error, det(V) is the determinant of V, and Trace(V) is the trace of V.

85 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/286,410Filed 5 Oct 2016 · us provisional application
Priority: 5 Oct 2015
Published: US 2017/0098312 A16 Apr 2017
Granted: 16 Jan 2018
Term ends: Not estimatedThe source records don’t support an estimate.

Classification25 codes

G02B 27/0172
Head mounted: characterised by optical features
A63F 13/26
Output arrangements for video game devices: having at least one additional display device, e.g. on the game controller or outside a game booth
G02B 27/017
Head-up displays: head mounted
G06T 7/246
Analysis of motion: using feature-based methods, e.g. the tracking of corners or segments