# US 12,008,922 B2 · Systems and methods for comparing driving performance for simulated driving

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,922 B2
- Document ID: US12008922B2
- Status: Granted
- Kind: B2
- Application number: 16/025,799
- Filed: 2018-07-02
- Published: 2020-01-02
- Granted: 2024-06-11
- Earliest claimed: 2018-07-02
- Assignees: SmartDrive Systems, Inc.
- Applicants: SmartDrive Systems, Inc.
- Inventors: Jason Palmer; Reza Ghanbari; Nicholas Shayne Brookins; Slaven Sljivar; Mark Freitas; Barry James Parshall; Daniel Andrew Deninger; Behzad Shahrasbi; Mojtaba Ziyadi
- Web page: https://patentgrep.com/patent/US12008922B2

## CPC

- G09B 9/052: characterised by provision for recording or measuring trainee's performance (devices for psychotechnics, e.g. for vehicle drivers A61B5/16, A61B5/18)
- B60W 40/09: Driving style or behaviour
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- G08G 1/167: {Driving aids for lane monitoring, lane changing, e.g. blind spot detection}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

This disclosure relates to a system that determines driving performance by a vehicle operator for simulated driving of a simulated vehicle in a simulation engine. Individual vehicle event scenarios correspond to vehicle events. Individual simulation scenarios correspond to individual vehicle event scenarios. A vehicle operator, e.g., an autonomous driving algorithm, operates the simulated vehicle in the simulation engine for a set of simulation scenarios. One or more metrics quantify the performance of the vehicle operator based on simulated results.

## Claims

### Claim 1

A system configured to determine driving performance by a vehicle operator and by a second vehicle operator for simulated driving of a simulated vehicle in a simulation engine, wherein the vehicle operator and the second vehicle operator are included in a set of vehicle operators, wherein the simulated driving is based on real-world operation of real-world vehicles, the system comprising:
electronic storage configured to electronically store information; and
one or more processors configured via machine-readable instructions to:
determine one or more vehicle parameters of individual vehicles, wherein the one or more vehicle parameters are determined directly based on output signals from at least two different sensors, wherein the one or more vehicle parameters are determined multiple times in an ongoing manner during operation of the vehicle;
obtain a stored set of vehicle event scenarios that correspond to vehicle events that have occurred in the real world at particular times during the real-world operation of the real-world vehicles, wherein individual vehicle events are associated with physical surroundings of individual vehicles around the particular times the individual vehicle events occurred, wherein individual vehicle events are detected based on comparing the one or more vehicle parameters of individual vehicles with one or more thresholds, wherein a first vehicle event scenario is associated with a first set of circumstances that is based on a first set of physical surroundings of a first vehicle around a particular time a first vehicle event occurred, wherein the first vehicle event scenario has a scenario time period that begins prior to an occurrence of a potential vehicle event;
create a set of simulation scenarios that are suitable for use by the simulation engine, wherein individual ones of the set of simulation scenarios correspond to individual ones of the stored set of vehicle event scenarios, wherein individual ones of the set of simulation scenarios mimic circumstances associated with a corresponding vehicle event scenario, such that a first simulation scenario mimics the first set of circumstances associated with the first vehicle event scenario, wherein the simulated vehicle is based on the first vehicle;
establish a communication link between the vehicle operator and the simulation engine, wherein the vehicle operator is an autonomous driving algorithm that controls operations of the simulated vehicle autonomously through the communication link;
run the set of simulation scenarios in the simulation engine, wherein the simulated vehicle interacts with the vehicle operator and is operated by the vehicle operator based on input received from the autonomous driving algorithm;
detect, by the simulation engine during the running of the set of simulation scenarios, one or more simulated vehicle events that have occurred to the simulated vehicle as the autonomous driving algorithm controlled the operations of the simulated vehicle;
determine one or more metrics that quantify a performance of the vehicle operator in running the set of simulation scenarios, wherein the determination of the one or more metrics is based on the one or more simulated vehicle events as detected;
establish a second communication link between the second vehicle operator and the simulation engine, wherein the second vehicle operator is a second autonomous driving algorithm that controls operations of the simulated vehicle autonomously through the second communication link;
run the set of simulation scenarios in the simulation engine, wherein the simulated vehicle interacts with the second vehicle operator and is operated by the second vehicle operator based on second input received from the second autonomous driving algorithm;
detect, by the simulation engine during the running of the set of simulation scenarios, a second set of simulated vehicle events that have occurred to the simulated vehicle as the second autonomous driving algorithm controlled the operations of the simulated vehicle;
determine a second set of metrics that quantify a second performance of the second vehicle operator running the set of simulation scenarios, wherein the determination of the second set of metrics is based on the second set of simulated vehicle events as detected;
compare the performance of the vehicle operator with the second performance of the second vehicle operator, wherein comparison is based on the one or more metrics and the second set of metrics;
based on the comparison, determine whether the vehicle operator or the second vehicle operator performed best in running the set of simulation scenarios; and
generate a report based on the comparison and transfer the report to one or more users.

### Claim 2 (depends on 1)

The system of claim 1, wherein the potential vehicle event corresponds to the first vehicle event.

### Claim 3 (depends on 1)

The system of claim 1, wherein the second vehicle operator is a benchmark autonomous driving algorithm.

### Claim 4 (depends on 1)

The system of claim 1, wherein individual ones of the stored set of vehicle event scenarios correspond to detected vehicle events in real life.

### Claim 5 (depends on 1)

The system of claim 1. wherein the communication link provides the vehicle operator with control over the operations of the simulated vehicle.

### Claim 6 (depends on 1)

The system of claim 1, wherein the simulation engine is computer-implemented and configured by a set of machine-readable instructions, and wherein the set of simulation scenarios in the simulation engine is run at faster-than-real-time.

### Claim 7 (depends on 1)

The system of claim 1, wherein one of the one or more metrics is reduced responsive to an individual one of the set of simulation scenarios resulting in an accident.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more processors are further configured to take an action based on the comparison of the performance of the vehicle operator with the second performance of the second vehicle operator.

### Claim 9

A method for determining driving performance by a vehicle operator and by a second vehicle operator for simulated driving of a simulated vehicle in a simulation engine, wherein the vehicle operator and the second vehicle operator are included in a set of vehicle operators, wherein the simulated driving is based on real-world operation of real-world vehicles, the method comprising:
determining one or more vehicle parameters of individual vehicles, wherein the one or more vehicle parameters are determined directly based on output signals from at least two different sensors, wherein the one or more vehicle parameters are determined multiple times in an ongoing manner during operation of the vehicle;
obtaining a stored set of vehicle event scenarios that correspond to vehicle events that have occurred in the real world at particular times during the real-world operation of the real-world vehicles, wherein individual vehicle events are associated with physical surroundings of individual vehicles around the particular times the individual vehicle events occurred, wherein individual vehicle events are detected based on comparing the one or more vehicle parameters of individual vehicles with one or more thresholds, wherein a first vehicle event scenario is associated with a first set of circumstances that is based on a first set of physical surroundings of a first vehicle around a particular time a first vehicle event occurred, wherein the first vehicle event scenario has a scenario time period that begins prior to an occurrence of a potential vehicle event;
creating a set of simulation scenarios that are suitable for use by the simulation engine, wherein individual ones of the set of simulation scenarios correspond to individual ones of the stored set of vehicle event scenarios, wherein individual ones of the set of simulation scenarios mimic circumstances associated with a corresponding vehicle event scenario, such that a first simulation scenario mimics the first set of circumstances associated with the first vehicle event scenario, wherein the simulated vehicle is based on the first vehicle;
establishing a communication link between the vehicle operator and the simulation engine, wherein the vehicle operator is an autonomous driving algorithm that controls operations of the simulated vehicle autonomously through the communication link;
running the set of simulation scenarios in the simulation engine, wherein the simulated vehicle interacts with the vehicle operator and is operated by the vehicle operator based on input received from the autonomous driving algorithm;
detecting, by the simulation engine during the running of the set of simulation scenarios, one or more simulated vehicle events that have occurred to the simulated vehicle as the autonomous driving algorithm controlled the operations of the simulated vehicle;
determining one or more metrics that quantify a performance of the vehicle operator in running the set of simulation scenarios, wherein the determination of the one or more metrics is based on the one or more simulated vehicle events as detected;
establishing a second communication link between the second vehicle operator and the simulation engine, wherein the second vehicle operator is a second autonomous driving algorithm that controls operations of the simulated vehicle autonomously through the second communication link;
running the set of simulation scenarios in the simulation engine, wherein the simulated vehicle interacts with the second vehicle operator and is operated by the second vehicle operator based on second input received from the second autonomous driving algorithm;
detecting, by the simulation engine during the running of the set of simulation scenarios, a second set of simulated vehicle events that have occurred to the simulated vehicle as the second autonomous driving algorithm controlled the operations of the simulated vehicle;
determining a second set of metrics that quantify a second performance of the second vehicle operator in running the set of simulation scenarios, wherein the determination of the second set of metrics is based on the second set of simulated vehicle events as detected;
comparing the performance of the vehicle operator with the second performance of the second vehicle operator, wherein comparison is based on the one or more metrics and the second set of metrics;
based on the comparison, determine whether the vehicle operator or the second vehicle operator performed best in running the set of simulation scenarios; and
generating a report based on the comparison and transferring the report to one or more users.

### Claim 10 (depends on 9)

The method of claim 9, wherein the potential vehicle event corresponds to the first vehicle event.

### Claim 11 (depends on 9)

The method of claim 9, wherein the second vehicle operator is a benchmark autonomous driving algorithm.

### Claim 12 (depends on 9)

The method of claim 9, wherein individual ones of the stored set of vehicle event scenarios correspond to detected vehicle events in real life.

### Claim 13 (depends on 9)

The method of claim 9, wherein the communication link provides the vehicle operator with control over the operations of the simulated vehicle.

### Claim 14 (depends on 9)

The method of claim 9, wherein the set of simulation scenarios in the simulation engine is run at faster-than-real-time.

### Claim 15 (depends on 9)

The method of claim 9, wherein one of the one or more metrics is reduced responsive to an individual one of the set of simulation scenarios resulting in an accident.

### Claim 16

The method of daim 9, wherein the report includes at least one of a warning and a recommendation reaarding the vehicle operator, and wherein at least one of the warning and the recommendation are based on at least one of the one or more metrics and the comparison.