# US 2025/0153353 A1 · Replaying and learning from events in robotics systems and applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0153353 A1
- Document ID: US20250153353A1
- Status: Published application
- Kind: A1
- Application number: 18/509,124
- Filed: 2023-11-14
- Published: 2025-05-15
- Earliest claimed: 2023-11-14
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: David J. ELLIOTT; Gordon GRIGOR; Arun Shamanna LAKSHMI
- Web page: https://patentgrep.com/patent/US20250153353A1

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion}
- B25J 11/008: {Manipulators for service tasks}
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

Systems and techniques to improve a robotics control application are described herein. In at least one embodiment, event information is generated by one or more robots performing a mission. The event information indicates one or more determinations made by the one or more robots. The event information is stored in a log, and the stored event information can be used to simulate the generating of decisions to control the one or more robots.

## Claims

### Claim 1

A method comprising:
obtaining event information generated using one or more robots performing a mission, the event information indicating one or more determinations made using the one or more robots and based at least on sensor data;
storing the event information in a log; and
performing one or more operations to simulate generating one or more decisions to control the one or more robots based at least on the event information stored in the log.

### Claim 2 (depends on 1)

The method of claim 1, wherein the performing the one or more operations to simulate the generating the one or more decisions comprises simulating the event information being received again.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more determinations include one or more objects being detected based at least on the sensor data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the event information indicates at least one of one or more actions performed using the one or more robots, one or more states of the one or more robots, or one or more reasons that one or more events occurred.

### Claim 5 (depends on 1)

The method of claim 1, wherein the obtaining and storing operations are performed using a communication interface between the one or more robots and a robotics control application that generates the one or more decisions to control the one or more robots.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more operations to simulate the generating the one or more decisions begins from an initial state of the one or more robots.

### Claim 7 (depends on 1)

The method of claim 1, wherein the one or more operations to simulate the generating the one or more decisions begins from a state of the one or more robots subsequent to an initial state of the one or more robots.

### Claim 8 (depends on 1)

The method of claim 1, further comprising modifying a robotics control application that generates the one or more decisions based at least on the event information.

### Claim 9 (depends on 1)

The method of claim 1, further comprising performing one or more operations to train a machine learning model based at least on the event information.

### Claim 10 (depends on 1)

The method of claim 1, further comprising detecting an error during the mission based at least on the event information and a trained machine learning model.

### Claim 11 (depends on 1)

The method of claim 1, further comprising receiving additional event information generated using one or more actors participating in the mission, wherein the one or more actors are not robots.

### Claim 12 (depends on 1)

The method of claim 1, wherein the method is performed by a processor comprised in at least one of:
an infotainment system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using the one or more robots;
a system for generating or presenting virtual reality, augmented reality, or mixed reality content;
a system for performing one or more conversational AI operations;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for performing one or more generative AI operations;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 13

A method comprising:
receiving, at a communication interface between one or more robots and a robotics control application, event information indicating one or more determinations made using the one or more robots and based at least on sensor data;
storing the event information in a log; and
transmitting the event information that is stored in the log to the communication interface during one or more simulations.

### Claim 14 (depends on 13)

The method of claim 13, wherein the one or more determinations include one or more objects being detected based at least on sensor data.

### Claim 15 (depends on 13)

The method of claim 13, wherein the event information indicates at least one of one or more actions performed using the one or more robots, one or more states of the one or more robots, or one or more reasons that one or more events occurred.

### Claim 16 (depends on 13)

The method of claim 13, further comprising modifying the robotics control application based at least on the event information.

### Claim 17 (depends on 13)

The method of claim 13, further comprising performing one or more operations to train a machine learning model to detect errors based at least on the event information.

### Claim 18 (depends on 13)

The method of claim 13, further comprising performing one or more operations to train a machine learning model to generate missions for controlling one or more robots based at least on the event information.

### Claim 19 (depends on 18)

The method of claim 18, wherein the machine learning model comprises a large language model (LLM).

### Claim 20

A system comprising:
one or more processors to perform one or more operations to simulate logged event information being received at a communication interface between one or more robots and a robotics control application, wherein the logged event information indicates one or more determinations made by the one or more robots based at least on sensor data.
