# US 10,121,259 B2 · System and method for determining motion and structure from optical flow

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,121,259 B2
- Document ID: US10121259B2
- Status: Granted
- Kind: B2
- Application number: 15/174,980
- Filed: 2016-06-06
- Published: 2017-01-26
- Granted: 2018-11-06
- Earliest claimed: 2015-06-04
- Assignees: New York University Langone Medical
- Applicants: New York University Langone Medical Center
- Inventors: Davi Geiger
- Web page: https://patentgrep.com/patent/US10121259B2

## CPC

- G06T 7/579: from motion
- G06T 2207/10016: Video; Image sequence
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method and system for extracting motion and structure from a sequence of images stored on a computer system, the method comprises obtaining images including a set of three-dimensional (3D) points over a plurality of frames, determining an instantaneous motion of the set of 3D points by an angular velocity and a translation with respect to an axis of rotation, computing an optical flow using the instantaneous motion of the set of 3D points based on a projection of velocity of the set of 3D points, computing a depth of the set of 3D points from the optical flow, and determining an epipolar line in the images using the optical flow.

## Claims

### Claim 1

A method for extracting motion and structure from a sequence of images stored on a computer system, the method comprising:
obtaining images including a set of three-dimensional (3D) points over a plurality of frames;
determining an instantaneous motion of the set of 3D points by an angular velocity and a translation with respect to an axis of rotation;
computing an optical flow using the instantaneous motion of the set of 3D points based on a projection of velocity of the set of 3D points;
computing a depth of the set of 3D points from the optical flow; and
determining an epipolar line in the images using the optical flow.

### Claim 2 (depends on 1)

The method of claim 1 wherein obtaining the images includes retrieving the images from a memory device of the computer system.

### Claim 3 (depends on 1)

The method of claim 1 further comprising analyzing, by a processor of the computer system, distances of object surfaces within the images based on the depth and the epipolar line in the images.

### Claim 4 (depends on 1)

The method of claim 1 wherein determining the instantaneous motion of the set of 3D points includes calculating V(P)=w×P+T, where V(P) is the instantaneous motion of a given point, w is the angular velocity, P is the given point, and T is a translation of the given point.

### Claim 5 (depends on 1)

The method of claim 1 wherein computing the optical flow comprises computing displacement of 3D velocity for the set of 3D points over the plurality of frames.

### Claim 6 (depends on 1)

The method of claim 1 wherein computing the optical flow comprises calculating
v
⁡
(
x
)
=
[
v
x
⁡
(
x
)
v
x
⁡
(
x
)
0
]
=
[
dx
dt
dy
dt
0
]
=
[
d
⁡
(
X
/
Z
)
dt
d
⁡
(
Y
/
Z
)
ⅆ
t
0
]
=
1
Z
2
⁡
[
ZV
x
-
XV
z
ZV
y
-
YV
z
0
]
=
1
Z
⁡
(
x
)
⁢
(
x
×
V
)
×
Z
^
,
where v(x) is an optical flow of a x coordinate point for a 3D plane including X, Y, Z coordinates, and V is an instantaneous motion for a given coordinate of a 3D point.

### Claim 7 (depends on 1)

The method of claim 1 wherein determining the epipolar line includes calculating
0
=
l
x
·
x
t
=
x
⁡
[
0
-
T
z
T
y
T
z
0
-
T
x
-
T
y
T
x
0
]
⁢
x
t
,
⁢
where
⁢
⁢
l
x
=
-
t
⁡
(
x
×
T
)
+
t
2
⁡
(
w
t
⁢
QT
)
⁢
Z
^
,
⁢
Q
=
[
x
]
2
=
[
-
(
1
+
y
2
)
xy
x
xy
-
(
1
+
x
2
)
y
x
y
-
(
x
2
+
y
2
)
]
,
t is a given time, w is the angular velocity, and T is a translation of a given coordinate.

### Claim 8 (depends on 1)

The method of claim 1 wherein the computer system is selected from a group consisting of robotic vision, self-driving vehicles, image coding, and 3D modeling.

### Claim 9

A system for extracting motion and structure from a sequence of images stored on a memory device, the system comprising:
a processor; and
a memory having executable instructions stored thereon that when executed by the processor cause the processor to:
obtain images including a set of three-dimensional (3D) points over a plurality of frames;
determine an instantaneous motion of the set of 3D points by an angular velocity and a translation with respect to an axis of rotation;
compute an optical flow using the instantaneous motion of the set of 3D points based on a projection of velocity of the set of 3D points;
compute a depth of the set of 3D points from the optical flow; and
determine an epipolar line in the images using the optical flow.

### Claim 10 (depends on 9)

The system of claim 9 further comprising the processor analyzing distances of object surfaces within the images based on the depth and the epipolar line in the images.

### Claim 11 (depends on 9)

The system of claim 9 wherein the processor determines the instantaneous motion of the set of 3D points includes the processor calculating V(P)=w×P+T, where V(P) is the instantaneous motion of a given point, w is the angular velocity, P is the given point, and T is a translation of the given point.

### Claim 12 (depends on 9)

The system of claim 9 wherein the processor computes the optical flow comprises the processor computing displacement of 3D velocity for the set of 3D points over the plurality of frames.

### Claim 13 (depends on 9)

The system of claim 9 wherein the processor computes the optical flow comprises the processor calculating
v
⁡
(
x
)
=
[
v
x
⁡
(
x
)
v
x
⁡
(
x
)
0
]
=
[
dx
dt
dy
dt
0
]
=
[
d
⁡
(
X
/
Z
)
dt
d
⁡
(
Y
/
Z
)
dt
0
]
=
1
Z
2
⁡
[
ZV
x
-
XV
z
ZV
y
-
YV
z
0
]
=
1
Z
⁡
(
x
)
⁢
(
x
×
V
)
×
Z
^
,
where v(x) is an optical flow of a x coordinate point for a 3D plane including X, Y, Z coordinates, and V is an instantaneous motion for a given coordinate of a 3D point.

### Claim 14 (depends on 9)

The system of claim 9 wherein the processor determines the epipolar line includes the processor calculating
0
=
l
x
·
x
t
=
x
⁡
[
0
-
T
z
T
y
T
z
0
-
T
x
-
T
y
T
x
0
]
⁢
x
t
,
⁢
where
⁢
⁢
l
x
=
-
t
⁡
(
x
×
T
)
+
t
2
⁡
(
w
t
⁢
QT
)
⁢
Z
^
,
⁢
Q
=
[
x
]
2
=
[
-
(
1
+
y
2
)
xy
x
xy
-
(
1
+
x
2
)
y
x
y
-
(
x
2
+
y
2
)
]
,
t is a given time, w is the angular velocity, and T is a translation of a given coordinate.

### Claim 15 (depends on 9)

The system of claim 9 wherein the depth of the set of 3D points and the epipolar line in the images are utilized in a system selected from a group consisting of robotic vision, self-driving vehicles, image coding, and 3D modeling.

### Claim 16

Non-transitory computer readable medium comprising program code that when executed by a programmable processor causes execution of a method for extracting motion and structure from a sequence of images stored on a computer system, the computer readable medium comprising:
computer program code for obtaining images including a set of three-dimensional (3D) points over a plurality of frames;
computer program code for determining an instantaneous motion of the set of 3D points by an angular velocity and a translation with respect to an axis of rotation;
computer program code for computing an optical flow using the instantaneous motion of the set of 3D points based on a projection of velocity of the set of 3D points;
computer program code for computing a depth of the set of 3D points from the optical flow; and
computer program code for determining an epipolar line in the images using the optical flow.

### Claim 17 (depends on 16)

The computer readable medium of claim 16 wherein the computer program code for obtaining the images includes computer program code for retrieving the images from a memory device of the computer system.

### Claim 18 (depends on 16)

The computer readable medium of claim 16 further comprising computer program code for analyzing, by a processor of the computer system, distances of object surfaces within the images based on the depth and the epipolar line in the images.

### Claim 19 (depends on 16)

The computer readable medium of claim 16 wherein the computer system is selected from a group consisting of robotic vision, self-driving vehicles, image coding, and 3D modeling.

### Claim 20 (depends on 16)

The computer readable medium of claim 16 wherein the computer program code for computing the optical flow further includes computer program code for computing displacement of 3D velocity for the set of 3D points over the plurality of frames.