# US 2017/0287108 A9 · Dense structure from motion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287108 A9
- Document ID: US20170287108A9
- Status: Published application
- Kind: A9
- Application number: 14/182,715
- Filed: 2014-02-18
- Published: 2017-10-05
- Earliest claimed: 2010-09-21
- Assignees: Mobileye Vision Technologies Ltd.
- Applicants: Mobileye Vision Technologies Ltd.
- Inventors: Harel Livyatan; Oded Berberian; Barak Cohen; Gideon Stein
- Web page: https://patentgrep.com/patent/US20170287108A9

## CPC

- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/579: from motion
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30256: Lane; Road marking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Determining three-dimensional structure in a road environment using a system mountable in a host vehicle including a camera connectible to a processor. Multiple image frames are captured in the field of view of the camera. In the image frames, a line is selected below which the road is imaged. The line separates between upper images essentially excluding images of the road and lower images essentially including images of the road. One or more of the lower images is warped, according to a road homography to produce at least one warped lower image. The three-dimensional structure may be provided from motion of a matching feature within the upper images or from motion of a matching feature within at least one of the lower images and at least one warped lower image.

## Claims

### Claim 1

A method for determining three-dimensional structure in a road environment, the method performed by a system mountable in a host vehicle, wherein the system includes a camera operatively connectible to a processor, the method comprising:
capturing a plurality of image frames in the field of view of the camera;
in the image frames selecting a line below which the road is imaged, wherein the line separates between upper images essentially excluding images of the road and lower images essentially including images of the road; and
warping at least one of the lower images according to a road homography to produce at least one warped lower image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
warping at least one of the image frames according to ego-motion of the host vehicle.

### Claim 3 (depends on 1)

The method of claim 1, further comprising.
providing the three-dimensional structure from motion of a matching feature within the upper images.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
providing the three-dimensional structure from motion of a matching feature within at least one of the lower images and said at least one warped lower image.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
first matching a feature between said lower images which are unwarped to produce a first matching result;
second matching said feature between at least one of said unwarped lower images and said at least one warped lower image to produce a second matching result; and
selecting an image disparity for said feature, corresponding to a matching result selected from the group consisting of said first matching result and said second matching result.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
from a disparity map of said at least one warped lower image, determining which image points are of the road;
selecting an image disparity for said image points of the road from matching features of said at least one warped lower image.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
pre-warping an image portion in the image frames corresponding to a vertical plane perpendicular to the direction of travel at a distance Z from the vehicle, wherein the distance Z is selectably either a fixed value or dependent on vehicle motion.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
pre-warping an image portion in the image frames corresponding to two vertical planes parallel to the direction of travel at a fixed distance to the left and right of the camera.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
said capturing at least three image frames in the field of view of the camera of the road environment, a first image frame at time A, a second image frame at time B and a third image frame at time C, wherein the time interval (C-A) is significantly greater than the time interval (B-A), wherein an epipole is a stationary point on image transformation between the at least three image frames;
for an image feature near the epipole, computing image motion between the third image frame captured at time C and the first image frame captured at time A;
for an image feature far from the epipole, computing image motion from the second image frame captured at time B and the first image frame captured.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
reducing image resolution of the image frames by performing a local histogram equalization around a matching feature within the image frames ;
providing the three-dimensional structure from motion of said matching feature within the image frames of reduced resolution.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
performing a rank transform on the image frames prior to computing motion disparity between the image frames, wherein the rank transform replaces a gray-scale of a picture element with a count of pixels in a local neighborhood of the picture element which have a lower gray-scale value than the picture element.

### Claim 12

A system for determining three-dimensional structure in a road environment, the system mountable in a host vehicle, including a camera operatively connectible to a processor, the system operable to:
capture a plurality of image frames in the field of view of the camera;
in the image frames, select a line below which the road is imaged, wherein the line separates between upper images essentially excluding images of the road and lower images essentially including images of the road; and
warp at least one of the lower images according to a road homography to produce at least one warped lower image.

### Claim 13 (depends on 12)

The system of claim 12, further operable to:
warp at least one of the image frames according to ego-motion of the host vehicle.

### Claim 14 (depends on 12)

The system of claim 12, further operable to:
provide the three-dimensional structure from motion of a matching feature within the upper images.

### Claim 15 (depends on 12)

The system of claim 12, further operable to:
provide the three-dimensional structure from motion of a matching feature within at least one of the lower images and said at least one warped lower image.
