# US 9,437,001 B2 · Tracking objects in bowl-shaped imaging systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,437,001 B2
- Document ID: US9437001B2
- Status: Granted
- Kind: B2
- Application number: 14/490,369
- Filed: 2014-09-18
- Published: 2016-03-24
- Granted: 2016-09-06
- Earliest claimed: 2014-09-18
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Kay-Ulrich Scholl; Koba Natroshvili
- Web page: https://patentgrep.com/patent/US9437001B2

## CPC

- G06T 7/73: using feature-based methods
- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- G06T 3/047: Fisheye or wide-angle transformations
- G06T 3/12: Panospheric to cylindrical image transformations
- G06T 7/536: from perspective effects, e.g. by using vanishing points
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- B60R 2300/10: characterised by the type of camera system used
- B60R 2300/30: characterised by the type of image processing
- B60R 2300/303: using joined images, e.g. multiple camera images
- B60R 2300/80: characterised by the intended use of the viewing arrangement
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Technologies for determining a distance of an object from a vehicle include a computing device to identify an object captured in a fisheye image generated by a fisheye camera of the vehicle. The computing device projects a contour of the identified object on a selected virtual plane that is located outside the vehicle and selected from a predefined set of virtual planes based on a location of the identified object relative to the vehicle. The computing device identifies a bottom of the projected contour on the selected virtual plane and determines an intersection point of an imaginary line with a ground plane coincident with a plane on which the vehicle is positioned. The imaginary line passes through each of the identified bottom of the projected contour and the fisheye camera. The computing device determines a location of the identified object relative to the vehicle based on the determined intersection point and the identified bottom of the projected contour.

## Claims

### Claim 1

A computing device for determining a distance of an object from a vehicle, the computing device comprising:
a fisheye camera to capture a fisheye image;
an object projection module to (i) identify an object captured in the fisheye image and (ii) project a contour of the identified object on a selected virtual plane that is located outside the vehicle and selected from a predefined set of virtual planes based on a location of the identified object relative to the vehicle; and
a location determination module to (i) identify a bottom of the projected contour on the selected virtual plane, (ii) determine an intersection point of an imaginary line with a ground plane coincident with a plane on which the vehicle is positioned, wherein the imaginary line passes through each of the identified bottom of the projected contour and the fisheye camera, and (iii) determine a location of the identified object relative to the vehicle based on the determined intersection point and the identified bottom of the projected contour.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein to determine the location of the identified object relative to the vehicle comprises to apply a Kalman filter based on the determined intersection point and the identified bottom of the projected contour.

### Claim 3 (depends on 2)

The computing device of claim 2, wherein to apply the Kalman filter comprises to:
determine a state vector of the Kalman filter according to {right arrow over (x)}=[xp yp zb]T;
determine a measurement vector of the Kalman filter according to {right arrow over (z)}=[xb yb zb]T;
apply the Kalman filter based on the determined state vector and the determined measurement vector, wherein:
xp, yp , and zp , are three-dimensional coordinates of the intersection point; and
xb, yb, and zb are three-dimensional coordinates of the bottom of the identified object.

### Claim 4 (depends on 3)

The computing device of claim 3, wherein to apply the Kalman filter comprises to:
determine a Jacobian according to
∂
z
∂
x
=
[
1
-
z
b
z
a
0
-
1
z
a
⁢
(
x
p
-
x
a
)
0
1
-
z
b
z
a
-
1
z
a
⁢
(
y
p
-
y
a
)
0
0
1
]
;
and
apply an extended Kalman filter based on the determined state vector, the determined measurement vector, and the determined Jacobian,
wherein xa, ya, and za are three-dimensional coordinates of the fisheye camera.

### Claim 5 (depends on 1)

The computing device of claim 1, wherein to project the contour of the identified object comprises to select a virtual plane that is nearest the identified object and on which to project the contour of the identified object.

### Claim 6 (depends on 1)

The computing device of claim 1, wherein to project the contour of the identified object comprises to:
determine a reference line intersecting the fisheye camera and a location of the identified object; and
project the contour of the identified object to a virtual plane that the reference line intersects.

### Claim 7 (depends on 1)

The computing device of claim 1, wherein to identify the bottom of the projected contour comprises to:
determine a bounding box of the projected contour on the virtual plane; and
identify a middle point of a bottom of the bounding box.

### Claim 8 (depends on 7)

The computing device of claim 7, wherein the intersection point is an intersection point of a center of the fisheye camera with the middle point of the bottom of the bounding box.

### Claim 9 (depends on 7)

The computing device of claim 7, wherein the bounding box is a smallest rectangular box bounding the projected contour.

### Claim 10 (depends on 1)

The computing device of claim 1, wherein to determine the intersection point of the line with the ground plane comprises to determine the intersection point according to
x
p
=
-
z
b
z
a
-
z
b
⁢
x
a
+
z
a
z
a
-
z
b
⁢
x
b
,
y
p
=
-
z
b
z
a
-
z
b
⁢
y
a
+
z
a
z
a
-
z
b
⁢
y
b
,
and zp=0, wherein:
xp, yp, and zp are three-dimensional coordinates of the intersection point;
xa, ya, and za are three-dimensional coordinates of the fisheye camera; and
xb, yb, and zb are three-dimensional coordinates of the bottom of the identified object.

### Claim 11 (depends on 1)

The computing device of claim 1, wherein the predefined set of virtual planes comprises at least twice as many virtual planes as there are fisheye cameras of the vehicle.

### Claim 12 (depends on 1)

The computing device of claim 1, wherein each virtual plane of the predefined set of virtual planes is orthogonal to the ground plane.

### Claim 13 (depends on 1)

The computing device of claim 1, wherein:
the predefined set of virtual planes consists of eight virtual planes;
a first and second virtual plane of the eight virtual planes are parallel to a side of the vehicle;
a third and fourth virtual plane of the eight virtual planes are parallel to a rear of the vehicle and orthogonal to the side of the vehicle;
a fifth and sixth virtual plane of the eight virtual planes are parallel to one another and at forty-five degree angles with respect to the first and third virtual planes;
a seventh and eighth virtual plane of the eight virtual planes are parallel to one another, orthogonal to the fifth and sixth virtual planes, and at forty-five degree angles with respect to the first and third virtual planes; and
each virtual plane of the predefined set of virtual planes is orthogonal to the ground plane.

### Claim 14 (depends on 1)

The computing device of claim 1, wherein the computing device is embodied as an in-vehicle computing system.

### Claim 15

One or more non-transitory machine-readable storage media comprising a plurality of instructions stored thereon that, in response to execution by a computing device, cause the computing device to:
identify an object captured in a fisheye image generated by a fisheye camera of the vehicle;
project a contour of the identified object on a selected virtual plane that is located outside the vehicle and selected from a predefined set of virtual planes based on a location of the identified object relative to the vehicle;
identify a bottom of the projected contour on the selected virtual plane;
determine an intersection point of an imaginary line with a ground plane coincident with a plane on which the vehicle is positioned, wherein the imaginary line passes through each of the identified bottom of the projected contour and the fisheye camera; and
determine a location of the identified object relative to the vehicle based on the determined intersection point and the identified bottom of the projected contour.

### Claim 16 (depends on 15)

The one or more non-transitory machine-readable storage media of claim 15, wherein to determine the location of the identified object relative to the vehicle comprises to:
determine a state vector of a Kalman filter according to {right arrow over (x)}=[xp yp zb]T;
determine a measurement vector of the Kalman filter according to {right arrow over (z)}=[xb yb zb]T; and
apply the Kalman filter based on the determined state vector and the determined measurement vector,
wherein xp, yp, and zp are three-dimensional coordinates of the determined intersection point and xb, yb, and zb are three-dimensional coordinates of the identified bottom of the projected contour.

### Claim 17 (depends on 16)

The one or more non-transitory machine-readable storage media of claim 16, wherein to apply the Kalman filter further comprises to:
determine a Jacobian according to
∂
z
∂
x
=
[
1
-
z
b
z
a
0
-
1
z
a
⁢
(
x
p
-
x
a
)
0
1
-
z
b
z
a
-
1
z
a
⁢
(
y
p
-
y
a
)
0
0
1
]
;
and
apply an extended Kalman filter based on the determined state vector, the determined measurement vector, and the determined Jacobian,
wherein xa, ya, and za are three-dimensional coordinates of the fisheye camera.

### Claim 18 (depends on 15)

The one or more non-transitory machine-readable storage media of claim 15, wherein to project the contour of the identified object comprises to select a virtual plane that is nearest the identified object and on which to project the contour of the identified object.

### Claim 19 (depends on 15)

The one or more non-transitory machine-readable storage media of claim 15, wherein to identify the bottom of the projected contour comprises to (i) determine a bounding box of the projected contour on the virtual plane and (ii) identify a middle point of a bottom of the bounding box; and
wherein to determine the intersection point of the line with the ground plane comprises to determine an intersection point of a center of the fisheye camera with the middle point of the bottom of the bounding box.

### Claim 20 (depends on 15)

The one or more non-transitory machine-readable storage media of claim 15, wherein to determine the intersection point of the line with the plane ground comprises to determine the intersection point according to
x
p
=
-
z
b
z
a
-
z
b
⁢
x
a
+
z
a
z
a
-
z
b
⁢
x
b
,
y
p
=
-
z
b
z
a
-
z
b
⁢
y
a
+
z
a
z
a
-
z
b
⁢
y
b
,
and zp=0 wherein:
xp, yp, and zp are three-dimensional coordinates of the intersection point;
xa, ya, and za are three-dimensional coordinates of the fisheye camera; and
xb, yb, and zb are three-dimensional coordinates of the bottom of the identified object.

### Claim 21

A method for determining a distance of an object from a vehicle by a computing device of the vehicle, the method comprising:
identifying, by the computing device, an object captured in a fisheye image generated by a fisheye camera of the vehicle;
projecting, by the computing device, a contour of the identified object on a selected virtual plane that is located outside the vehicle and selected from a predefined set of virtual planes based on a location of the identified object relative to the vehicle;
identifying, by the computing device, a bottom of the projected contour on the selected virtual plane;
determining, by the computing device, an intersection point of an imaginary line with a ground plane coincident with a plane on which the vehicle is positioned, wherein the imaginary line passes through each of the identified bottom of the projected contour and the fisheye camera; and
determining, by the computing device, a location of the identified object relative to the vehicle based on the determined intersection point and the identified bottom of the projected contour.

### Claim 22 (depends on 21)

The method of claim 21, wherein determining the location of the identified object relative to the vehicle comprises applying a Kalman filter based on the determined intersection point and the identified bottom of the projected contour by:
determining a state vector of the Kalman filter according to {right arrow over (x)}=[xp yp zb]T;
determining a measurement vector of the Kalman filter according to {right arrow over (z)}=[xb yb zb]T;
applying the Kalman filter based on the determined state vector and the determined measurement vector, wherein:
xp, yp, and zp are three-dimensional coordinates of the intersection point; and
xb, yband zb are three-dimensional coordinates of the bottom of the identified object.

### Claim 23 (depends on 22)

The method of claim 22, wherein applying the Kalman filter further comprises:
determining a Jacobian according to
∂
z
∂
x
=
[
1
-
z
b
z
a
0
-
1
z
a
⁢
(
x
p
-
x
a
)
0
1
-
z
b
z
a
-
1
z
a
⁢
(
y
p
-
y
a
)
0
0
1
]
;
and
applying an extended Kalman filter based on the determined state vector, the determined measurement vector, and the determined Jacobian,
wherein xa, ya, and za are three-dimensional coordinates of the fisheye camera.

### Claim 24 (depends on 21)

The method of claim 21, wherein identifying the bottom of the projected contour comprises (i) determining a bounding box of the projected contour on the virtual plane and (ii) identifying a middle point of a bottom of the bounding box; and
wherein determining the intersection point of the line with the ground plane comprises determining an intersection point of a center of the fisheye camera with the middle point of the bottom of the bounding box.

### Claim 25 (depends on 21)

The method of claim 21, wherein determining the intersection point of the line with the ground plane comprises determining the intersection point according to
x
p
=
-
z
b
z
a
-
z
b
⁢
x
a
+
z
a
z
a
-
z
b
⁢
x
b
,
y
p
=
-
z
b
z
a
-
z
b
⁢
y
a
+
z
a
z
a
-
z
b
⁢
y
b
,
and zp=0 , wherein:
xp, yp, and zp are three-dimensional coordinates of the intersection point;
xa, ya, and za are three-dimensional coordinates of the fisheye camera; and
xb, yb, and zb are three-dimensional coordinates of the bottom of the identified object.