# US 2021/0276587 A1 · Systems and Methods for Autonomous Vehicle Systems Simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0276587 A1
- Document ID: US20210276587A1
- Status: Published application
- Kind: A1
- Application number: 17/150,982
- Filed: 2021-01-15
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Uber Technologies, Inc.
- Applicants: Uber Technologies, Inc.
- Inventors: Raquel Urtasun; Kelvin Ka Wing Wong; Qiang Zhang; Bin Yang; Ming Liang; Renjie Liao
- Web page: https://patentgrep.com/patent/US20210276587A1

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- G06F 11/0739: {in a data processing system embedded in automotive or aircraft systems}
- B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces}
- G06F 11/3013: {where the computing system is an embedded system, i.e. a combination of hardware and software dedicated to perform a certain function in mobile devices, printers, automotive or aircraft systems (testing or monitoring of control systems or parts thereof G05B23/02)}
- G06F 11/3652: {in-circuit-emulation [ICE] arrangements}
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F 30/15: Vehicle, aircraft or watercraft design
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- B60W 2050/0019: {Control system elements or transfer functions}
- B60W 2050/0083: {Setting, resetting, calibration}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods of the present disclosure are directed to a method. The method can include obtaining simplified scenario data associated with a simulated scenario. The method can include determining, using a machine-learned perception-prediction simulation model, a simulated perception-prediction output based at least in part on the simplified scenario data. The method can include evaluating a loss function comprising a perception loss term and a prediction loss term. The method can include adjusting one or more parameters of the machine-learned perception-prediction simulation model based at least in part on the loss function.

## Claims

### Claim 1

A computing system, comprising:
one or more processors;
a machine-learned perception-prediction simulation model configured to simulate a perception system and a prediction system of an autonomous vehicle, comprising:
a perception submodel configured to process simplified scenario data to obtain simulated perception data; and
a prediction submodel configured to process the simplified scenario data and the simulated perception data to obtain simulated prediction data; and
one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining the simplified scenario data associated with a simulated scenario for the perception system and the prediction system of the autonomous vehicle;
determining, using the machine-learned perception-prediction simulation model, a simulated perception-prediction output based at least in part on the simplified scenario data, wherein the simulated perception-prediction output comprises the simulated perception data and the simulated prediction data;
providing the simulated perception-prediction output to a motion planning system of the autonomous vehicle; and
in response to providing the simulated perception-prediction output, obtaining a motion plan output from the motion planning system of the autonomous vehicle.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the operations further comprise determining a performance metric associated with the motion plan output.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the motion planning system comprises one or more machine-learned motion planning models; and
wherein the operations further comprise:
evaluating a loss function that evaluates a difference between the motion plan output and a ground truth label;
determining, based at least in part on the loss function, one or more parameter adjustments for at least one of the one or more machine-learned motion planning models; and
providing the one or more parameter adjustments to the motion planning system.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein:
the computing system comprises a service entity computing system; and
the autonomous vehicle is configured to provide one or more services facilitated by the service entity.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein the computing system comprises an autonomous vehicle computing system of the autonomous vehicle.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein determining, using the machine-learned perception-prediction simulation model, the simulated perception-prediction output based at least in part on the simplified scenario data comprises:
determining, using the perception submodel of the machine-learned perception-prediction simulation model, the simulated perception data based at least in part on the simplified scenario data; and
determining, using the prediction submodel of the machine-learned perception-prediction simulation model, the simulated prediction data based at least in part on the simplified scenario data and the simulated perception data.

### Claim 7 (depends on 6)

The computing system of claim 6, wherein the simulated perception data comprises one or more simulated bounding boxes for one or more respective predicted entities of the simplified scenario data; and
wherein determining, using the prediction submodel of the machine-learned perception-prediction simulation model, the simulated prediction data based at least in part on the simplified scenario data and the simulated perception data comprises:
determining one or more feature vectors respectively descriptive of the one or more simulated bounding boxes; and
determining, using the prediction submodel of the machine-learned perception-prediction simulation model, the simulated prediction data based at least in part on the one or more feature vectors, wherein the simulated prediction data comprises one or more future state predictions for each of the one or more simulated bounding boxes.

### Claim 8

A computer-implemented method, comprising:
obtaining, by a computing system comprising one or more computing devices, simplified scenario data associated with a simulated scenario;
determining, by the computing system using a machine-learned perception-prediction simulation model, a simulated perception-prediction output based at least in part on the simplified scenario data, wherein the simulated perception-prediction output comprises simulated perception data and simulated prediction data, wherein the machine-learned perception-prediction simulation model is configured to simulate a perception system and a prediction system of an autonomous vehicle;
evaluating, by the computing system, a loss function comprising a perception loss term and a prediction loss term, wherein the perception loss term evaluates a difference between the simulated perception data and ground truth perception data and the prediction loss term evaluates a difference between the simulated prediction data and ground truth prediction data; and
adjusting, by the computing system, one or more parameters of the machine-learned perception-prediction simulation model based at least in part on the loss function.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein determining, by the computing system using the machine-learned perception-prediction simulation model, the simulated perception-prediction output based at least in part on the simplified scenario data comprises:
determining, by the computing system using a perception submodel of the machine-learned perception-prediction simulation model, the simulated perception data based at least in part on the simplified scenario data; and
determining, by the computing system using a prediction submodel of the machine-learned perception-prediction simulation model, the simulated prediction data based at least in part on the simplified scenario data and the simulated perception data.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the simulated perception data comprises one or more simulated bounding boxes for one or more respective predicted objects of the simplified scenario data; and
wherein determining, by the computing system using the prediction submodel of the machine-learned perception-prediction simulation model, the simulated prediction data based at least in part on the simplified scenario data and the simulated perception data comprises:
determining, by the computing system, one or more feature vectors respectively descriptive of the one or more simulated bounding boxes; and
determining, by the computing system using the prediction submodel of the machine-learned perception-prediction simulation model, the simulated prediction data based at least in part on the one or more feature vectors, wherein the simulated prediction data comprises one or more future state predictions for each of the one or more simulated bounding boxes.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein:
the one or more bounding boxes comprises a plurality of bounding boxes; and
a future state prediction for a first simulated bounding box is based at least in part on a future state prediction of a second simulated bounding box of the plurality of simulated bounding boxes.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, wherein:
the perception submodel comprises a convolutional neural network; and
the prediction submodel comprises a multi-layer perceptron.

### Claim 13 (depends on 8)

The computer-implemented method of claim 8, wherein:
the ground truth perception data comprises a perception output of the perception system of the autonomous vehicle;
the ground truth prediction data comprises a prediction output of the prediction system of the autonomous vehicle; and
the perception output and the prediction output are based at least in part on sensor data associated with the simulated scenario.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein:
the perception output of the perception system comprises a false positive bounding box; and
at least one of the one or more simulated bounding boxes comprises a simulated false positive bounding box.

### Claim 15 (depends on 8)

The computer-implemented method of claim 8, wherein the operations further comprise:
obtaining, by the computing system, additional simplified scenario data associated with an additional simulated scenario;
determining, by the computing system using the machine-learned perception-prediction simulation model, an additional simulated perception output and an additional simulated prediction output based at least in part on the additional simplified scenario data; and
providing, by the computing system, the additional simulated perception output and the additional simulated prediction output to a motion planning system of the autonomous vehicle.

### Claim 16 (depends on 8)

The computer-implemented method of claim 8, wherein the simulated scenario represents an intermediate output associated with a scenario for a perception system and a prediction system of an autonomous vehicle.

### Claim 17 (depends on 8)

The computer-implemented method of claim 8, wherein the simplified scenario data comprises:
high-definition mapping data of an area of a transportation network associated with the simulated scenario;
one or more entity representations located within the area of the transportation network, wherein each of the one or more entity representations comprises an entity class label and an associated bounding box; and
metadata descriptive of at least a starting state and a planned route of the autonomous vehicle.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein, prior to determining the simulated perception-prediction output, the method comprises determining, by the computing system, one or more occlusion masks based at least in part on a location of at least the autonomous vehicle in the area of the transportation network; and
wherein the simplified scenario data further comprises the one or more occlusion masks.

### Claim 19

One or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining simplified scenario data that represents an intermediate output associated with a simulated scenario for a perception system and a prediction system of an autonomous vehicle;
processing the simplified scenario data with a perception submodel of a machine-learned perception-prediction simulation model to obtain simulated perception data, wherein the machine-learned perception-prediction simulation model is configured to simulate the perception system and the prediction system of the autonomous vehicle;
processing the simplified scenario data and the simulated perception data with a prediction submodel of the machine-learned perception-prediction simulation model to obtain simulated prediction data;
evaluating a loss function comprising a perception loss term and a prediction loss term, wherein the perception loss term evaluates a difference between the simulated perception data and ground truth perception data and the prediction loss term evaluates a difference between the simulated prediction data and ground truth prediction data; and
adjusting one or more parameters of the machine-learned perception-prediction simulation model based at least in part on the loss function.

### Claim 20 (depends on 19)

The one or more tangible, non-transitory media of claim 19, wherein the simulated perception data comprises one or more simulated bounding boxes for one or more respective predicted entities of the simplified scenario data; and
wherein processing, by the computing system, the simplified scenario data and the simulated perception output with the prediction submodel comprises:
determining one or more feature vectors respectively descriptive of the one or more simulated bounding boxes; and
processing the one or more feature vectors with the prediction submodel of the machine-learned perception-prediction simulation model to obtain the simulated prediction data, wherein the simulated prediction data comprises one or more future state predictions for each of the one or more simulated bounding boxes.
