# US 12,222,832 B2 · Systems and methods for generating synthetic sensor data via machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,222,832 B2
- Document ID: US12222832B2
- Status: Granted
- Kind: B2
- Application number: 18/466,286
- Filed: 2023-09-13
- Published: 2023-12-28
- Granted: 2025-02-11
- Earliest claimed: 2019-03-23
- Assignees: Aurora Operations, Inc.
- Applicants: Aurora Operations, Inc.
- Inventors: Sivabalan Manivasagam; Shenlong Wang; Wei-Chiu Ma; Kelvin Ka Wing Wong; Wenyuan Zeng; Raquel Urtasun
- Web page: https://patentgrep.com/patent/US12222832B2

## CPC

- G01S 7/497: Means for monitoring or calibrating
- G06F 11/263: Generation of test inputs, e.g. test vectors, patterns or sequences {; with adaptation of the tested hardware for testability with external testers}
- G01S 13/931: of land vehicles
- G01S 17/006: {Theoretical aspects}
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 15/06: Ray-tracing
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/05: Geographic models
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2210/56: Particle system, point based geometry or rendering
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

## Claims

### Claim 1

An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:
using a physics-based simulation engine to obtain an initial synthetic point cloud,
generating, by a machine-learned model, a value corresponding to a probability that a real LIDAR point cloud would have an error at a point in the initial synthetic point cloud, and
generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and
instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

### Claim 2 (depends on 1)

The autonomous vehicle control system of claim 1, wherein generating, based on the value, the synthetic LIDAR data to contain the error at the point comprises:
sampling, based on the value, to determine whether to include the error at the point in the synthetic LIDAR data.

### Claim 3 (depends on 1)

The autonomous vehicle control system of claim 1, wherein the probability corresponds to a dropout probability.

### Claim 4 (depends on 3)

The autonomous vehicle control system of claim 3, wherein generating the synthetic LIDAR data to contain the error at the point comprises omitting the point.

### Claim 5 (depends on 1)

The autonomous vehicle control system of claim 1, wherein the synthetic LIDAR data was generated by:
generating, using the machine-learned model, a probability mask over the initial synthetic point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and
sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

### Claim 6 (depends on 1)

The autonomous vehicle control system of claim 1, wherein the at least one of the one or more machine-learned models was tested by:
processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

### Claim 7 (depends on 6)

The autonomous vehicle control system of claim 6, wherein the at least one of the one or more machine-learned models was tested by:
generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and
processing the additional synthetic LiDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

### Claim 8 (depends on 6)

The autonomous vehicle control system of claim 6, wherein the at least one of the one or more machine-learned models was tested by:
inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.

### Claim 9

One or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:
using a physics-based simulation engine to obtain an initial synthetic point cloud,
generating, by a machine-learned model, a value corresponding to a probability that a real LiDAR point cloud would have an error at a point in the initial synthetic point cloud, and
generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and
instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising:
obtaining LIDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

### Claim 10 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein using the machine-learned model to add realistic errors to the initial synthetic point cloud comprises:
sampling, based on the value, to determine whether to include the error at the point in the synthetic LIDAR data and.

### Claim 11 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein the probability corresponds to a dropout probability.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein generating, based on the sampling, the synthetic LIDAR data to contain the error at the point comprises omitting the point.

### Claim 13 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein using the machine-learned model to add realistic errors to the initial synthetic point cloud comprises:
generating, using the machine-learned model, a probability mask over the initial synthetic point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and
sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

### Claim 14 (depends on 9)

The one or more non-transitory computer-readable media of claim 9, wherein the at least one of the one or more machine-learned models was tested by:
processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

### Claim 15 (depends on 14)

The one or more non-transitory computer-readable media of claim 14, wherein the at least one of the one or more machine-learned models was tested by:
generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and
processing the additional synthetic LiDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

### Claim 16 (depends on 14)

The one or more non-transitory computer-readable media of claim 14, wherein the at least one of the one or more machine-learned models was tested by:
inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.

### Claim 17

A method, comprising:
obtaining LiDAR data descriptive of an environment of an autonomous vehicle;
determining, using one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:
using a physics-based simulation engine to obtain an initial synthetic point cloud,
generating, by a machine-learned model, a value corresponding to a probability that a real LiDAR point cloud would have an error at a point in the initial synthetic point cloud,
generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and
controlling the autonomous vehicle according to the motion plan.

### Claim 18 (depends on 17)

The method of claim 17, wherein using the machine-learned model to add realistic errors to the initial point cloud comprises:
generating, using the machine-learned model, a probability mask over the initial point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and
sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

### Claim 19 (depends on 17)

The method of claim 17, wherein the at least one of the one or more machine-learned models was tested by:
processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

### Claim 20 (depends on 19)

The method of claim 19, wherein the at least one of the one or more machine-learned models was tested by:
generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and
processing the additional synthetic LIDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

### Claim 21

An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:
using a physics-based simulation engine to obtain an initial synthetic point cloud, and
using a machine-learned model to add realistic errors to the initial synthetic point cloud to obtain the synthetic LiDAR data;
wherein the at least one of the one or more machine-learned models was tested by processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data; and
instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

### Claim 22 (depends on 21)

The autonomous vehicle control system of claim 21, wherein the at least one of the one or more machine-learned models was tested by:
inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.