# US 12,112,504 B2 · Method, system, and computer program product for parallax estimation for sensors for autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,504 B2
- Document ID: US12112504B2
- Status: Granted
- Kind: B2
- Application number: 17/512,765
- Filed: 2021-10-28
- Published: 2023-05-04
- Granted: 2024-10-08
- Earliest claimed: 2021-10-28
- Assignees: Argo AI, LLC
- Applicants: Argo AI, LLC
- Inventors: Michel Laverne
- Web page: https://patentgrep.com/patent/US12112504B2

## CPC

- G01S 7/4972: {Alignment of sensor}
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- B60W 60/001: {Planning or execution of driving tasks}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 7/497: Means for monitoring or calibrating
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- H04N 17/002: {for television cameras}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- B60W 2050/146: {Display means}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- G01S 17/931: of land vehicles
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, systems, and products for parallax estimation for sensors for autonomous vehicles may include generating a two-dimensional grid based on a field of view of a first sensor. For each respective point in the grid, a three-dimensional position of an intersection point between a first ray from the first sensor and a second ray from a second sensor may be determined. For each respective intersection point, a respective solid angle may be determined based on a first three-dimensional vector from the first sensor and a second three-dimensional vector from the second sensor to the intersection point. A matrix may be generated based on a distance from the first sensor, a distance from the second sensor, and the solid angle for each respective intersection point. At least one metric may be extracted from the matrix. An arrangement of the first and second sensors may be adjusted based on the metric(s).

## Claims

### Claim 1

A method for parallax estimation for sensors for autonomous vehicles performed by a parallax estimation and sensor placement system, comprising:
generating a two-dimensional grid based on a field of view of a first sensor of an autonomous vehicle, the grid comprising a plurality of points;
for each respective point of the plurality of points, determining a three-dimensional position of a respective intersection point between a first respective ray from the first sensor to the respective point of the plurality of points and a second respective ray from a second sensor of the autonomous vehicle, wherein a plurality of intersection points comprises each respective intersection point for each respective point of the plurality of points;
for each respective intersection point of the plurality of intersection points, determining a respective solid angle based on a first three-dimensional vector from the first sensor to the three-dimensional position of the respective intersection point and a second three-dimensional vector from the second sensor to the three-dimensional position of the respective intersection point;
generating a matrix based on a first distance from the first sensor to each respective intersection point, a second distance from the second sensor to each respective intersection point, and the solid angle for each respective intersection point;
extracting at least one metric from the matrix;
adjusting an arrangement of the first sensor and the second sensor based on the at least one metric; interpolating the two-dimensional grid based on the at least one metric; and
displaying a visual representation based on interpolating the two-dimensional grid, wherein displaying the visual representation comprises displaying a gradient of at least one of shades or colors based on the at least one metric across a two-dimensional image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first sensor comprises an image capture system and the second sensor comprises a ray casting system.

### Claim 3 (depends on 2)

The method of claim 2, wherein the image capture system comprises a camera and the ray casting system comprises a LiDAR.

### Claim 4 (depends on 1)

The method of claim 1, further comprising receiving first sensor parameters for the first sensor, first position data associated with a first position of the first sensor, first orientation data associated with a first orientation of the first sensor, second sensor parameters for the second sensor, second position data associated with a second position of the second sensor, and second orientation data associated with a second orientation of the second sensor,
wherein the first sensor parameters comprise the field of view of the first sensor,
wherein the second sensor parameters comprise ray data associated with a plurality of rays cast by the second sensor, and
wherein adjusting the arrangement comprises adjusting at least one of the first position of the first sensor, the first orientation of the first sensor, the second position of the second sensor, the second orientation of the second sensor, or any combination thereof.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the two-dimensional grid comprises:
generating a plurality of horizontal lines across the field of view; and
generating a plurality of vertical lines across the field of view,
wherein each respective intersection between a respective horizontal line of the plurality of horizontal lines and a respective vertical line of the plurality of vertical lines comprises a respective point of the plurality of points.

### Claim 6 (depends on 1)

The method of claim 1, further comprising determining the first respective ray from the first sensor to each respective point of the plurality of points based on an azimuth and an elevation for each respective point with respect to the first sensor.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first sensor comprises a camera and the azimuth and the elevation for each respective point are with respect to a center of a lens of the camera.

### Claim 8 (depends on 1)

The method of claim 1, wherein the second respective ray comprises a worst-case ray of a plurality of rays cast by the second sensor.

### Claim 9 (depends on 8)

The method of claim 8, wherein the second sensor comprises a LiDAR, and wherein the worst-case ray comprises a lowest beam having a lowest elevation if the LiDAR is positioned above the first sensor or a highest beam having a highest elevation if the LiDAR is positioned below the first sensor.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
receiving boundary data associated with at least one boundary condition;
determining that a first intersection point of the plurality of intersection points does not satisfy the at least one boundary condition; and
replacing the first intersection point with a replacement intersection point between the first respective ray from the first sensor and a lowest remaining beam having a lowest remaining elevation if the LiDAR is positioned above the first sensor or a highest remaining beam having a highest remaining elevation if the LiDAR is positioned below the first sensor, wherein the replacement intersection point does satisfy the at least one boundary condition.

### Claim 11 (depends on 10)

The method of claim 10, wherein the at least one boundary condition comprises at least one of a first minimum range for the first sensor, a second minimum range for the second sensor, a first maximum range for the first sensor, a second maximum range for the second sensor, a ground surface position of a ground surface upon which the autonomous vehicle is located, or any combination thereof.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
generating the first three-dimensional vector based on the three-dimensional position of the respective intersection point and a first three-dimensional position of the first sensor; and
generating the second three-dimensional vector based on the three-dimensional position of the respective intersection point and a second three-dimensional position of the second sensor.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining the respective solid angle comprises:
generating a first unit vector based on the first three-dimensional vector;
generating a second unit vector based on the second three-dimensional vector; and
calculating the solid angle based on the first unit vector and the second unit vector.

### Claim 14 (depends on 1)

The method of claim 1, wherein generating the matrix comprises:
generating a 3×n matrix, wherein n is a number of intersection points of the plurality of intersection points; and
populating each respective column of the 3×n matrix with the first distance from the first sensor to the respective intersection point, the second distance from the second sensor to the respective intersection point, and the solid angle for the respective intersection point.

### Claim 15 (depends on 1)

The method of claim 1, wherein extracting the at least one metric from the matrix comprises at least one of:
determining a minimum solid angle based on the matrix;
determining a maximum solid angle based on the matrix;
determining a minimum range of solid angles based on the matrix;
determining a maximum range of solid angles based on the matrix;
determining a spread of solid angles based on the matrix;
determining an average of solid angles based on the matrix;
determining a standard deviation of solid angles based on the matrix;
determining a three-sigma range of solid angles based on the matrix;
determining a minimum first distance based on the matrix;
determining a maximum first distance based on the matrix;
determining a minimum range of first distances based on the matrix;
determining a maximum range of first distances based on the matrix;
determining a spread of first distances based on the matrix;
determining an average of first distances based on the matrix;
determining a standard deviation of first distances based on the matrix;
determining a three-sigma range of first distances based on the matrix;
determining a minimum second distance based on the matrix;
determining a maximum second distance based on the matrix;
determining a minimum range of second distances based on the matrix;
determining a maximum range of second distances based on the matrix;
determining a spread of second distances based on the matrix;
determining an average of second distances based on the matrix;
determining a standard deviation of second distances based on the matrix;
determining a three-sigma range of second distances based on the matrix; or
any combination thereof.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
receiving potential arrangement data associated with a plurality of potential arrangements of the first sensor and the second sensor, the plurality of potential arrangements comprising a first potential arrangement and a second potential arrangement,
wherein the first sensor and the second sensor are initially in the first potential arrangement, and
wherein adjusting the arrangement comprises:
adjusting the arrangement from the first potential arrangement to the second potential arrangement based on the at least one metric satisfying a threshold; and
repeating generating of the two-dimensional grid, determining of the three-dimensional position of the respective intersection point for each respective point of the plurality of points, determining the respective solid angle for each respective intersection point, generating the matrix, and extracting the at least one metric from the matrix based on the first sensor and the second sensor being in the second potential arrangement.

### Claim 17

A system for parallax estimation for sensors for autonomous vehicles, comprising:
an autonomous vehicle;
a first sensor of the autonomous vehicle;
a second sensor of the autonomous vehicle; and
a parallax estimation and sensor placement system configured to:
generate a two-dimensional grid based on a field of view of the first sensor of the autonomous vehicle, the grid comprising a plurality of points;
for each respective point of the plurality of points, determine a three-dimensional position of a respective intersection point between a first respective ray from the first sensor to the respective point of the plurality of points and a second respective ray from the second sensor of the autonomous vehicle, wherein a plurality of intersection points comprises each respective intersection point for each respective point of the plurality of points;
for each respective intersection point of the plurality of intersection points, determine a respective solid angle based on a first three-dimensional vector from the first sensor to the three-dimensional position of the respective intersection point and a second three-dimensional vector from the second sensor to the three-dimensional position of the respective intersection point;
generate a matrix based on a first distance from the first sensor to each respective intersection point, a second distance from the second sensor to each respective intersection point, and the solid angle for each respective intersection point; and
extract at least one metric from the matrix; interpolate the two-dimensional grid based on the at least one metric; and
display a visual representation based on interpolating the two-dimensional grid, wherein the visual representation comprises a gradient of at least one of shades or colors based on the at least one metric across a two-dimensional image,
wherein an arrangement of the first sensor and the second sensor is adjusted based on the at least one metric.

### Claim 18

A computer program product for parallax estimation for a first sensor and a second sensor of an autonomous vehicle, the computer program product comprising at least one non-transitory computer-readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to:
generate a two-dimensional grid based on a field of view of the first sensor of the autonomous vehicle, the grid comprising a plurality of points;
for each respective point of the plurality of points, determine a three-dimensional position of a respective intersection point between a first respective ray from the first sensor to the respective point of the plurality of points and a second respective ray from the second sensor of the autonomous vehicle, wherein a plurality of intersection points comprises each respective intersection point for each respective point of the plurality of points;
for each respective intersection point of the plurality of intersection points, determine a respective solid angle based on a first three-dimensional vector from the first sensor to the three-dimensional position of the respective intersection point and a second three-dimensional vector from the second sensor to the three-dimensional position of the respective intersection point;
generate a matrix based on a first distance from the first sensor to each respective intersection point, a second distance from the second sensor to each respective intersection point, and the solid angle for each respective intersection point; and
extract at least one metric from the matrix; interpolate the two-dimensional grid based on the at least one metric; and
display a visual representation based on interpolating the two-dimensional grid, wherein the visual representation comprises a gradient of at least one of shades or colors based on the at least one metric across a two-dimensional image,
wherein an arrangement of the first sensor and the second sensor is adjusted based on the at least one metric.