# US 10,872,433 B2 · Vehicle environment modeling with a camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,433 B2
- Document ID: US10872433B2
- Status: Granted
- Kind: B2
- Application number: 16/387,969
- Filed: 2019-04-18
- Published: 2019-10-24
- Granted: 2020-12-22
- Earliest claimed: 2018-04-18
- Assignees: Mobileye Vision Technologies Ltd.
- Applicants: Mobileye Vision Technologies Ltd.
- Inventors: Gideon Stein; Itay Blumenthal; Nadav Shaag; Jeffrey Moskowitz
- Web page: https://patentgrep.com/patent/US10872433B2

## CPC

- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- B60W 40/06: Road conditions
- G06T 7/248: {involving reference images or patches}
- G06T 7/55: from multiple images
- G06T 7/579: from motion
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/165: {for passive traffic, e.g. including static obstacles, trees}
- B60W 2420/403: Image sensing, e.g. optical camera
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G08G 1/09623: {Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle (G08G1/0967 takes precedence)}
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

System and techniques for vehicle environment modeling with a camera are described herein. A time-ordered sequence of images representative of a road surface may be obtained. An image from this sequence is a current image. A data set may then be provided to an artificial neural network (ANN) to produce a three-dimensional structure of a scene. Here, the data set includes a portion of the sequence of images that includes the current image, motion of the sensor from which the images were obtained, and an epipole. The road surface is then modeled using the three-dimensional structure of the scene.

## Claims

### Claim 1

A device for modeling a road surface, the device comprising:
a hardware interface to obtain a time-ordered sequence of images representative of a road surface, the time-ordered sequence of images captured by a sensor, one of the sequence of images being a current image; and
processing circuitry to:
provide a data set to an artificial neural network (ANN), the ANN trained with an unsupervised training technique in which error is determined by a difference in road structure of overlapping segments between two images at two different times from the sequence of images, wherein the inference is performed on the first image, and wherein the overlapping segment is closer to the sensor in the second image, the ANN to produce a three-dimensional structure of a scene, the data set including:
a portion of the sequence of images, the portion of the sequence of images including the current image;
motion of the sensor that captured the sequence of images; and
an epipole; and
model the road surface using the three-dimensional structure of the scene.

### Claim 2 (depends on 1)

The device of claim 1, wherein the epipole is provided as a gradient image with a same dimensionality as the current image, values of pixels in the gradient image representing a distance from the epipole of pixels in the current image.

### Claim 3 (depends on 2)

The device of claim 2, wherein the gradient image represents only horizontal distances from the epipole, and wherein a second gradient image is provided to the ANN to represent vertical distances from the epipole.

### Claim 4 (depends on 1)

The device of claim 1, wherein the motion of the sensor is provided as a constant value image with a same dimensionality as the current image.

### Claim 5 (depends on 4)

The device of claim 4, wherein the constant value is a ratio of forward motion of the sensor by a height of the sensor from the plane.

### Claim 6 (depends on 1)

The device of claim 1, wherein, to model the road surface, the processing circuitry identifies a reflective area by comparing the three-dimensional structure of the scene with output from a second ANN, the second ANN trained to accept the portion of the sequence of images and produce a second three-dimensional structure, wherein training of the second ANN used more photogrammetric loss in the portion of the sequence of images than training the first ANN.

### Claim 7 (depends on 1)

The device of claim 1, wherein the processing circuitry is configured to invoke a second ANN using the three-dimensional structure to determine whether the features represent an object moving or not moving within an environment of the road surface.

### Claim 8 (depends on 1)

The device of claim 1, wherein the ANN is a convolutional neural network (CNN).

### Claim 9 (depends on 8)

The device of claim 8, wherein the motion of the sensor and the epipole are provided to the CNN at a bottleneck layer.

### Claim 10 (depends on 1)

The device of claim 1, wherein the ANN is trained with an unsupervised training technique in which error is determined by measuring a difference between a model of a future image and the future image, the model of the future image produced via a gamma warping of an image previous to the future image.

### Claim 11 (depends on 1)

The device of claim 1, wherein the road structure is represented in gamma.

### Claim 12

A method for modeling a road surface, the method comprising:
obtaining a time-ordered sequence of images representative of a road surface, the time-ordered sequence of images captured by a sensor, one of the sequence of images being a current image;
providing a data set to an artificial neural network (ANN), the ANN trained with an unsupervised training technique in which error is determined by a difference in road structure of overlapping segments between two images at two different times from the sequence of images, wherein the inference is performed on the first image, and wherein the overlapping segment is closer to the sensor in the second image, the ANN to produce a three-dimensional structure of a scene, the data set including:
a portion of the sequence of images, the portion of the sequence of images including the current image;
motion of the sensor that captured the sequence of images; and
an epipole; and
modeling the road surface using the three-dimensional structure of the scene.

### Claim 13 (depends on 12)

The method of claim 12, wherein the epipole is provided as a gradient image with a same dimensionality as the current image, values of pixels in the gradient image representing a distance from the epipole of pixels in the current image.

### Claim 14 (depends on 13)

The method of claim 13, wherein the gradient image represents only horizontal distances from the epipole, and wherein a second gradient image is provided to the ANN to represent vertical distances from the epipole.

### Claim 15 (depends on 12)

The method of claim 12, wherein the motion of the sensor is provided as a constant value image with a same dimensionality as the current image.

### Claim 16 (depends on 15)

The method of claim 15, wherein the constant value is a ratio of forward motion of the sensor by a height of the sensor from the plane.

### Claim 17 (depends on 12)

The method of claim 12, wherein modeling the road surface includes identifying a reflective area by comparing the three-dimensional structure of the scene with output from a second ANN, the second ANN trained to accept the portion of the sequence of images and produce a second three-dimensional structure, wherein training of the second ANN used more photogrammetric loss in the portion of the sequence of images than training the first ANN.

### Claim 18 (depends on 12)

The method of claim 12, comprising invoking a second ANN using the three-dimensional structure to determine whether the features represent an object moving or not moving within an environment of the road surface.

### Claim 19 (depends on 12)

The method of claim 12, wherein the ANN is a convolutional neural network (CNN).

### Claim 20 (depends on 19)

The method of claim 19, wherein the motion of the sensor and the epipole are provided to the CNN at a bottleneck layer.

### Claim 21 (depends on 12)

The method of claim 12, wherein the ANN is trained with an unsupervised training technique in which error is determined by measuring a difference between a model of a future image and the future image, the model of the future image produced via a gamma warping of an image previous to the future image.

### Claim 22 (depends on 12)

The method of claim 12, wherein the road structure is represented in gamma.