# US 2021/0278523 A1 · Systems and Methods for Integrating Radar Data for Improved Object Detection in Autonomous Vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278523 A1
- Document ID: US20210278523A1
- Status: Published application
- Kind: A1
- Application number: 17/150,590
- Filed: 2021-01-15
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Raquel Urtasun; Bin Yang; Ming Liang; Sergio Casas; Runsheng Benson Guo
- Web page: https://patentgrep.com/patent/US20210278523A1

## CPC

- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 7/417: {involving the use of neural networks}
- G01S 13/58: Velocity or trajectory determination systems; Sense-of-movement determination systems
- G01S 13/589: {measuring the velocity vector}
- G01S 13/931: of land vehicles
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
- G06F 18/251: {of input or preprocessed data}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/803: {of input or preprocessed data}
- G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for integrating radar and LIDAR data are disclosed. In particular, a computing system can access radar sensor data and LIDAR data for the area around the autonomous vehicle. The computing system can determine, using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle. The computing system can, for a respective object, select a plurality of radar points from the radar sensor data. The computing system can generate a similarity score for each selected radar point. The computing system can generate weight associated with each radar point based on the similarity score. The computing system can calculate predicted velocity for the respective object based on a weighted average of a plurality of velocities associated with the plurality of radar points. The computing system can generate a proposed motion plan based on the predicted velocity for the respective object.

## Claims

### Claim 1

A computer-implemented method for generating perception and prediction data for autonomous vehicles, the method comprising:
accessing, by a computing system including one or more processors, radar sensor data for an area around an autonomous vehicle, the radar sensor data including a plurality of radar points;
accessing, by the computing system, LIDAR sensor data for the area around the autonomous vehicle;
generating, by the computing system using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data;
determining, by the computing system using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle based on the fused representation;
for a respective object in the one or more objects:
selecting a plurality of radar points from the radar sensor data;
generating, by the computing system using a machine-learned model, a similarity score for each selected radar point;
generating, by the computing system, a weight associated with each radar point based, at least in part, on the similarity score; and
calculating, by the computing system, a predicted velocity for the respective object based on a weighted average of a plurality of velocities associated with the plurality of radar points; and
generating, by the computing system, a proposed motion plan based on the predicted velocity for the respective object.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the radar sensor data includes, for each radar point, a location of the radar point and a velocity associated with the radar point.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the radar sensor data includes data from a plurality of cycles of the radar sensor.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the LIDAR sensor data includes a plurality of LIDAR points, each LIDAR point having an associated location.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the LIDAR sensor data includes data from a plurality of sweeps of the LIDAR sensor.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein generating, by the computing system using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data further comprises:
generating, by the computing system, a voxel grid representation of the radar sensor data; and
generating, by the computing system, a voxel grid representation of the LIDAR sensor data.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the voxel grid representation of the radar sensor data includes a plurality of voxels and each voxel is associated with a voxel occupancy value.

### Claim 8 (depends on 7)

The computer-implemented method claim 7, wherein the voxel occupancy value for a respective voxel is based on a number of radar points that fall within the area associated with the respective voxels.

### Claim 9 (depends on 6)

The computer-implemented method of claim 6, wherein generating, by the computing system using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data further comprises:
identifying, by the computing system using the voxel grid representation of the radar sensor data as input to one or more machine-learned models, feature data for the radar sensor data; and
identifying, by the computing system using the voxel grid representation of the LIDAR sensor data as input to one or more machine-learned models, feature data for the LIDAR sensor data.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein generating, by the computing system using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data further comprises:
concatenating, by the computing system, feature data associated with the radar sensor data and feature data associated with the LIDAR sensor data and associating the concatenated data with a corresponding point in the fused representation.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
filtering, by the computing system, the radar sensor data to remove radar points that have an associated velocity value below a predetermined threshold.

### Claim 12

A computing system for generating perception and prediction data for autonomous vehicles, the system comprising:
one or more processors and one or more non-transitory computer-readable memories;
wherein the one or more non-transitory computer-readable memories store instructions that, when executed by the processor, cause the computing system to perform operations, the operations comprising:
accessing radar sensor data for an area around an autonomous vehicle, the radar sensor data including a plurality of radar points;
accessing LIDAR sensor data for the area around the autonomous vehicle;
generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR data;
determining, using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle based on the fused representation;
for a respective object in the one or more objects:
selecting a plurality of radar points from the radar sensor data;
generating, using a machine-learned model, a similarity score for each selected radar point;
generating a weight associated with each radar point based on the similarity score; and
calculating a predicted velocity for the respective object based on a weighted average of the velocities associated with the plurality of radar points; and
generating a proposed motion plan based on the predicted velocity for the respective object.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein selecting a plurality of radar points from the radar sensor data further comprises:
determining, by the computing system, for each radar point in the plurality of radar points, a distance from the radar point to a point associated with the respective object; and
selecting, by the computing system, one or more radar points based on the distance between determined distance between each radar point and a point associated with the respective object.

### Claim 14 (depends on 13)

The computing system of claim 13, wherein the point associated with the respective object is an estimated center of the object.

### Claim 15 (depends on 12)

The computing system of claim 12, wherein the respective object has an estimated direction.

### Claim 16 (depends on 15)

The computing system of claim 15, wherein generating a similarity score for each selected radar point further comprises:
generating, by the computing system, for a radar point, a modified velocity value based on the estimated direction of the respective object and the velocity associated with the radar point.

### Claim 17 (depends on 16)

The computing system of claim 16, wherein the modified velocity value is parallel to the estimated direction of the respective object.

### Claim 18 (depends on 12)

The computing system of claim 12, wherein generating, by the computing system, a weight associated with each radar point based on the similarity score further comprises:
normalizing, by the computing system, the weights associated with the plurality of radar points.

### Claim 19 (depends on 12)

The computing system of claim 12, further comprising:
transmitting vehicle motion controls to one or more vehicle control systems to implement the motion plan.

### Claim 20

An autonomous vehicle, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
accessing radar sensor data for an area around an autonomous vehicle, the radar sensor data including a plurality of radar points;
accessing LIDAR sensor data for the area around the autonomous vehicle;
generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR data;
determining, using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle based on the fused representation;
for a respective object in the one or more objects:
selecting a plurality of radar points from the radar sensor data;
generating, using a machine-learned model, a similarity score for each selected radar point;
generating a weight associated with each radar point based on the similarity score; and
calculating a predicted velocity for the respective object based on a weighted average of the velocities associated with the plurality of radar points; and
generating a proposed motion plan based on the predicted velocity for the respective object.
