# US 12,223,744 B2 · Vehicle environment modeling with a camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,744 B2
- Document ID: US12223744B2
- Status: Granted
- Kind: B2
- Application number: 17/777,212
- Filed: 2021-03-01
- Published: 2024-05-30
- Granted: 2025-02-11
- Earliest claimed: 2021-03-01
- Assignees: Mobileye Vision Technologies Ltd.
- Applicants: Mobileye Vision Technologies Ltd.
- Inventors: Gideon Stein; Itay Blumenthal; Jeffrey Moskowitz; Nadav Shaag; Natalie Carlebach
- Web page: https://patentgrep.com/patent/US12223744B2

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- B60W 40/06: Road conditions
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/50: Depth or shape recovery
- G06T 17/05: Geographic models
- G06V 10/20: Image preprocessing
- B60W 2420/403: Image sensing, e.g. optical camera
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

System and techniques for vehicle environment modeling with a camera are described herein. A device for modeling an environment comprises: a hardware sensor interface to obtain a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image; and processing circuitry to: provide the first unrectified image, the previous unrectified image, and the previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; determine a selected homography; and apply the selected homography to the three-dimensional structure of the scene to create a model of the road environment.

## Claims

### Claim 1

A method for modeling an environment, comprising:
obtaining a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image;
calculating a best-fit homography from a pinhole homography and rectification parameters for a lens used to capture the first unrectified image;
warping the previous unrectified image and the previous-previous unrectified image to the first unrectified image using the best-fit homography, to obtain a warped previous unrectified image and a warped previous-previous unrectified image;
providing the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; and
applying the best-fit homography to the three-dimensional structure of the scene to create a model of the road environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein the sequence of unrectified images is time-ordered.

### Claim 3 (depends on 1)

The method of claim 1, wherein the road environment includes a road surface.

### Claim 4 (depends on 1)

The method of claim 1, wherein the road environment includes a lane marker, a sidewalk, or a road control object.

### Claim 5 (depends on 1)

The method of claim 1, wherein calculating the best-fit homography comprises:
selecting a first set of image coordinates in the previous unrectified image;
rectifying the first set of image coordinates to obtain a first set of rectified image coordinates;
applying the pinhole homography to the first set of rectified image coordinates to identify corresponding image coordinates in the first unrectified image;
distorting the corresponding image coordinates in the first unrectified image to obtain estimated positions of distorted image coordinates; and
using a modeling technique to determine the best-fit homography between the first set of image coordinates and the estimated positions of distorted image coordinates.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first set of image coordinates are pixels.

### Claim 7 (depends on 5)

The method of claim 5, wherein the first set of image coordinates includes 50-200 image coordinates.

### Claim 8 (depends on 5)

The method of claim 5, wherein distorting the corresponding image coordinates comprises using a lens distortion model for a lens used to capture the first unrectified image, the previous unrectified image, and the previous-previous unrectified image.

### Claim 9 (depends on 5)

The method of claim 5, wherein the modeling technique is a linear regression technique.

### Claim 10 (depends on 1)

The method of claim 1, wherein the model of the road environment includes height above a ground plane and depth from an image capture device used to capture the sequence of unrectified images.

### Claim 11 (depends on 1)

The method of claim 1, wherein the ANN is trained with an unsupervised training technique that include operations to:
provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to the artificial neural network (ANN) to produce a gamma image;
rectify the gamma image to produce a rectified gamma image; and
calculate a loss function using the rectified gamma image.

### Claim 12 (depends on 11)

The method of claim 11, wherein to rectify the gamma image, the unsupervised training technique is to use a lens distortion model.

### Claim 13 (depends on 11)

The method of claim 11, wherein calculating the loss function comprises calculating an error between a model of a future image and a future image, the model of the future image produced using the best-fit homography applied to the first unrectified image.

### Claim 14

A device for modeling an environment, the device comprising:
a hardware sensor interface to obtain a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image; and
processing circuitry to:
calculate a best-fit homography from a pinhole homography and rectification parameters for a lens used to capture the first unrectified image;
warp the previous unrectified image and the previous-previous unrectified image to the first unrectified image using the best-fit homography, to obtain a warped previous unrectified image and a warped previous-previous unrectified image;
provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; and
apply the best-fit homography to the three-dimensional structure of the scene to create a model of the road environment.

### Claim 15 (depends on 14)

The device of claim 14, wherein the sequence of unrectified images is time-ordered.

### Claim 16 (depends on 14)

The device of claim 14, wherein the road environment includes a road surface.

### Claim 17 (depends on 14)

The device of claim 14, wherein the road environment includes a lane marker, a sidewalk, or a road control object.

### Claim 18 (depends on 14)

The device of claim 14, wherein to calculate the best-fit homography, the processing circuitry is to:
select a first set of image coordinates in the previous unrectified image;
rectify the first set of image coordinates to obtain a first set of rectified image coordinates;
apply the pinhole homography to the first set of rectified image coordinates to identify corresponding image coordinates in the first unrectified image;
distort the corresponding image coordinates in the first unrectified image to obtain estimated positions of distorted image coordinates; and
use a modeling technique to determine the best-fit homography between the first set of image coordinates and the estimated positions of distorted image coordinates.

### Claim 19 (depends on 18)

The device of claim 18, wherein the first set of image coordinates are pixels.

### Claim 20 (depends on 18)

The device of claim 18, wherein the first set of image coordinates includes 50-200 image coordinates.

### Claim 21 (depends on 18)

The device of claim 18, wherein to distort the corresponding image coordinates, the processing circuitry is to use a lens distortion model for a lens used to capture the first unrectified image, the previous unrectified image, and the previous-previous unrectified image.

### Claim 22 (depends on 18)

The device of claim 18, wherein the modeling technique is a linear regression technique.

### Claim 23 (depends on 14)

The device of claim 14, wherein the model of the road environment includes height above a ground plane and depth from an image capture device used to capture the sequence of unrectified images.

### Claim 24 (depends on 14)

The device of claim 14, wherein the ANN is trained with an unsupervised training technique that include operations to:
provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to the artificial neural network (ANN) to produce a gamma image;
rectify the gamma image to produce a rectified gamma image; and
calculate a loss function using the rectified gamma image.

### Claim 25 (depends on 24)

The device of claim 24, wherein to rectify the gamma image, the processing circuitry is to use a lens distortion model.

### Claim 26 (depends on 24)

The device of claim 24, wherein to calculate the loss function, the processing circuitry is to calculate an error between a model of a future image and a future image, the model of the future image produced using the best-fit homography applied to the first unrectified image.

### Claim 27

At least one non-transitory machine-readable medium including instructions for modeling an environment, the instructions, when executed by processing circuitry, cause the processing circuitry to perform operations comprising:
obtaining a sequence of unrectified images representative of a road environment, the sequence of unrectified images including a first unrectified image, a previous unrectified image, and a previous-previous unrectified image;
calculating a best-fit homography from a pinhole homography and rectification parameters for a lens used to capture the first unrectified image;
warping the previous unrectified image and the previous-previous unrectified image to the first unrectified image using the best-fit homography, to obtain a warped previous unrectified image and a warped previous-previous unrectified image;
providing the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to an artificial neural network (ANN) to produce a three-dimensional structure of a scene; and
applying the best-fit homography to the three-dimensional structure of the scene to create a model of the road environment.

### Claim 28 (depends on 27)

The non-transitory machine-readable medium of claim 27, wherein the sequence of unrectified images is time-ordered.

### Claim 29 (depends on 27)

The non-transitory machine-readable medium of claim 27, wherein the road environment includes a road surface.

### Claim 30 (depends on 27)

The non-transitory machine-readable medium of claim 27, wherein the road environment includes a lane marker, a sidewalk, or a road control object.

### Claim 31 (depends on 27)

The non-transitory machine-readable medium of claim 27, wherein the operations to calculate the best-fit homography, comprise operations to:
select a first set of image coordinates in the previous unrectified image;
rectify the first set of image coordinates to obtain a first set of rectified image coordinates;
apply the pinhole homography to the first set of rectified image coordinates to identify corresponding image coordinates in the first unrectified image;
distort the corresponding image coordinates in the first unrectified image to obtain estimated positions of distorted image coordinates; and
use a modeling technique to determine the best-fit homography between the first set of image coordinates and the estimated positions of distorted image coordinates.

### Claim 32 (depends on 31)

The non-transitory machine-readable medium of claim 31, wherein the first set of image coordinates are pixels.

### Claim 33 (depends on 31)

The non-transitory machine-readable medium of claim 31, wherein the first set of image coordinates includes 50-200 image coordinates.

### Claim 34 (depends on 31)

The non-transitory machine-readable medium of claim 31, wherein the operations to distort the corresponding image coordinates comprise operations to use a lens distortion model for a lens used to capture the first unrectified image, the previous unrectified image, and the previous-previous unrectified image.

### Claim 35 (depends on 31)

The non-transitory machine-readable medium of claim 31, wherein the modeling technique is a linear regression technique.

### Claim 36 (depends on 27)

The non-transitory machine-readable medium of claim 27, wherein the model of the road environment includes height above a ground plane and depth from an image capture device used to capture the sequence of unrectified images.

### Claim 37 (depends on 27)

The non-transitory machine-readable medium of claim 27, wherein the ANN is trained with an unsupervised training technique that include operations to:
provide the first unrectified image, the warped previous unrectified image, and the warped previous-previous unrectified image to the artificial neural network (ANN) to produce a gamma image;
rectify the gamma image to produce a rectified gamma image; and
calculate a loss function using the rectified gamma image.

### Claim 38 (depends on 37)

The non-transitory machine-readable medium of claim 37, wherein to rectify the gamma image, the processing circuitry is to use a lens distortion model.

### Claim 39 (depends on 37)

The non-transitory machine-readable medium of claim 37, wherein the operations to calculate the loss function comprise operations to calculate an error between a model of a future image and a future image, the model of the future image produced using the selected best-fit homography applied to the first unrectified image.