# US 12,182,694 B2 · Training, testing, and verifying autonomous machines using simulated environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,694 B2
- Document ID: US12182694B2
- Status: Granted
- Kind: B2
- Application number: 17/898,887
- Filed: 2022-08-30
- Published: 2023-01-05
- Granted: 2024-12-31
- Earliest claimed: 2018-03-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Clement Farabet; John Zedlewski; Zachary Taylor; Greg Heinrich; Claire Delaunay; Mark Daly; Matthew Campbell; Curtis Beeson; Gary Hicok; Michael Cox; Rev Lebaredian; Tony Tamasi; David Auld
- Web page: https://patentgrep.com/patent/US12182694B2

## CPC

- G06N 3/08: Learning methods
- G06F 9/455: Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 20/00: Machine learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In various examples, physical sensor data may be generated by a vehicle in a real-world environment. The physical sensor data may be used to train deep neural networks (DNNs). The DNNs may then be tested in a simulated environment—in some examples using hardware configured for installation in a vehicle to execute an autonomous driving software stack—to control a virtual vehicle in the simulated environment or to otherwise test, verify, or validate the outputs of the DNNs. Prior to use by the DNNs, virtual sensor data generated by virtual sensors within the simulated environment may be encoded to a format consistent with the format of the physical sensor data generated by the vehicle.

## Claims

### Claim 1

A method comprising:
receiving, using a simulator component, simulation data representing a simulated environment corresponding to a virtual representation of an autonomous or semi-autonomous machine;
generating, using the simulator component and based at least on the simulation data, virtual sensor data representing the simulated environment perceived using at least one virtual sensor of the virtual representation of the autonomous or semi-autonomous machine within the simulated environment;
receiving, using a computing system of a hardware-in-the-loop (HIL) device and from the simulator component, the virtual sensor data, the HIL device including hardware corresponding to the autonomous or semi-autonomous machine;
applying the virtual sensor data to one or more machine learning models being executed using the computing system of the HIL device;
determining, using the one or more machine learning models being executed using the computing system of the HIL device and based at least on the virtual sensor data, at least one operation for the virtual representation; and
transmitting, using the computing system and to the simulator component, operative data representing the at least one operation for the virtual representation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the virtual sensor data is generated using simulation software executed using one or more computing devices.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the computing system of the HIL device that includes the hardware corresponding to the autonomous or semi-autonomous machine is included in a vehicle simulator component; and
the vehicle simulator component communicates with the simulator component to receive the simulation data and transmit the operative data.

### Claim 4 (depends on 2)

The method of claim 2, wherein:
when integrated into an autonomous or semi-autonomous machine, the computing system communicates with one or more other components of the autonomous or semi-autonomous machine using at least one of a communication type or a communication protocol; and
during a simulation corresponding to the simulation data, the computing system communicates with the one or more computing devices using the at least one of the communication type or the communication protocol.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating encoded sensor data by encoding the virtual sensor data using a sensor data format that corresponds to real-world sensor data generated using a real-world sensor of the autonomous or semi-autonomous machine,
wherein the determining the at least one operation is based at least on the encoded sensor data.

### Claim 6 (depends on 1)

The method of claim 1, wherein the virtual sensor data includes a sensor data format that corresponds to real-world sensor data generated using a real-world sensor of the autonomous or semi-autonomous machine.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating, using the simulation data, additional virtual sensor data representing the simulated environment perceived by at least one additional virtual sensor of the virtual representation; and
determining, using the one or more machine learning models executed using the computing system of the HIL device, at least one additional operation for the virtual representation based at least on the additional virtual sensor data,
wherein the operative data further represents the at least one additional operation for the virtual representation.

### Claim 8 (depends on 1)

The method of claim 1, wherein the virtual representation comprises at least one of a virtual vehicle or a virtual robot that simulates the autonomous or semi-autonomous machine.

### Claim 9 (depends on 1)

The method of claim 1, wherein the simulation data includes virtual data and real-world data, and the simulated environment includes one or more representations corresponding to the virtual data and one or more representations corresponding to the real-world data.

### Claim 10

A method comprising:
obtaining virtual sensor data representing a simulated environment perceived by at least one virtual sensor of a virtual representation of an autonomous or semi-autonomous machine within the simulated environment, the virtual sensor data corresponding to a first data format;
generating, based at least on the virtual sensor data corresponding to the first data format, encoded sensor data by encoding the virtual sensor data using a second data format that is associated with at least one of one or more machine learning models or a hardware-in-the-loop (HIL) device that includes hardware corresponding to the autonomous or semi-autonomous machine;
determining, using the HIL device and based at least on the one or more machine learning models, at least one operation for the virtual representation based at least on the encoded sensor data; and
transmitting operative data representing the at least one operation of the virtual representation.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
the HIL device is included in a vehicle simulator component; and
the vehicle simulator component communicates with a simulation component to perform the transmitting of the operative data.

### Claim 12 (depends on 10)

The method of claim 10, wherein:
when in deployment within the autonomous or semi-autonomous machine, the hardware communicates with one or more other components of the autonomous or semi-autonomous machine using at least one of a communication type or a communication protocol; and
during simulation, the hardware communicates with one or more computing devices to obtain at least one of the virtual sensor data or encoded sensor data using the at least one of the communication type or the communication protocol.

### Claim 13 (depends on 10)

The method of claim 10, wherein the at least one of virtual sensor data or the encoded sensor data is generated using simulation software executed using a one or more computing devices.

### Claim 14 (depends on 10)

The method of claim 10, wherein the virtual representation corresponds to at least one of a virtual vehicle or a virtual robot that simulates the autonomous or semi-autonomous machine.

### Claim 15 (depends on 10)

The method of claim 10, wherein the simulated environment includes representations of virtual data augmented with representations of real-world data.

### Claim 16 (depends on 10)

The method of claim 10, wherein the second data format of the encoded sensor data is associated with a driving software stack that is executed using the HIL device that includes the hardware corresponding to the autonomous or semi-autonomous machine, the driving software stack including the one or more machine learning models.

### Claim 17 (depends on 10)

The method of claim 10, wherein:
the generating of the encoded sensor data by encoding the virtual sensor data is performed using a simulation component; and
the method further comprises receiving, using one or more computing devices associated with the HIL device, the virtual sensor data from the simulation component.

### Claim 18

A computing device comprising one or more processing units to:
receive, from one or more computing devices, virtual sensor data representing a simulated environment perceived using at least one virtual sensor of a virtual representation of an autonomous or semi-autonomous machine within the simulated environment, the computing device being part of a hardware-in-the-loop (HIL) device that includes hardware for installation in the autonomous or semi-autonomous machine;
determine, using one or more machine learning models, at least one operation for the virtual representation based at least on the virtual sensor data; and
transmit, to at least one computing device of the one or more computing devices, operative data representing the at least one operation of the virtual representation.

### Claim 19 (depends on 18)

The computing device of claim 18, wherein:
when installed in the autonomous or semi-autonomous machine, the hardware communicates with one or more other components of the autonomous or semi-autonomous machine using at least one of a communication type or a communication protocol; and
during simulation, the computing device communicates with the one or more computing devices using the at least one of the communication type or the communication protocol.

### Claim 20 (depends on 18)

The computing device of claim 18, wherein the computing device is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing real-time streaming;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.