# US 2025/0026365 A1 · Autonomous perception human-in-loop verification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0026365 A1
- Document ID: US20250026365A1
- Status: Published application
- Kind: A1
- Application number: 18/222,612
- Filed: 2023-07-17
- Published: 2025-01-23
- Earliest claimed: 2023-07-17
- Assignees: Mercedes-Benz Group AG
- Applicants: Mercedes-Benz Group AG
- Inventors: Aaron BROWN; Thomas MONNINGER; Vikram BHARADWAJ
- Web page: https://patentgrep.com/patent/US20250026365A1

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 19/003: {Navigation within 3D models or images}
- B60K 2360/175: Autonomous driving
- B60W 2050/146: {Display means}
- B60W 2540/00: Input parameters relating to occupants
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- B60W 2554/4029: Pedestrians
- B60W 2554/80: Spatial relation or speed relative to objects
- B60W 2555/00: Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
- G06T 2215/16: Using real world measurements to influence rendering
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES

## Abstract

A system can receive a perception output from an autonomous vehicle as the autonomous vehicle traverses a segment of a route. The perception output can be based on sensor data generated by a sensor suite of the autonomous vehicle. The system can generate, for an output device of a user, a human-perceptible three-dimensional rendering of an external environment of the vehicle based on the perception output. The human-perceptible three-dimensional rendering of the external environment can be as the user performs driving operations for the vehicle to traverse the segment of the route.

## Claims

### Claim 1

A non-transitory computer-readable medium that stores instructions, which when executed by one or more processors a computing system, cause the computing system to perform operations that include:
receiving a perception output from an autonomous vehicle as the autonomous vehicle traverses a segment of a route, the perception output being based on sensor data generated by a sensor suite of the autonomous vehicle; and
generating, for an output device of a user, a human-perceptible three-dimensional rendering of an external environment of the autonomous vehicle based on the perception output;
wherein generating the human-perceptible three-dimensional rendering of the external environment includes updating the human-perceptible three-dimensional rendering as the user performs driving operations for the autonomous vehicle to traverse the segment of the route.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the human-perceptible three-dimensional rendering of the external environment includes generating, for each of multiple objects of multiple types that are identified by the perception output, a corresponding representation that reflects one or more physical characteristics of the type of object.

### Claim 3 (depends on 2)

The non-transitory computer-readable medium of claim 2, wherein the multiple objects include one or more objects that are in motion, and wherein the corresponding representation of each object of the one or more objects is generated to be dynamic to reflect a motion of the object.

### Claim 4 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the multiple objects include a pedestrian that is on or near a roadway on which the autonomous vehicle operates.

### Claim 5 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the multiple objects include a second vehicle that is adjacent to or within proximity of the autonomous vehicle.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the human-perceptible three-dimensional rendering of the external environment includes generating a representation of a traffic sign or signal that is identified by the perception output.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the human-perceptible three-dimensional representation is generated in real-time while the user performs the driving operations from within the autonomous vehicle.

### Claim 8 (depends on 7)

The non-transitory computer-readable medium of claim 7, wherein the human-perceptible three-dimensional representation is rendered for a virtual reality headset worn by the user.

### Claim 9 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the human-perceptible three-dimensional representation is generated as a simulation for the user while the user performs the driving operations in a simulated environment.

### Claim 10 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein updating the human-perceptible three-dimensional rendering is based on at least one of the perception output or the driving operations.

### Claim 11

A computing system comprising:
one or more processors;
a memory to store instructions;
wherein the one or more processors execute the instructions stored in the memory to perform operations that include:
receiving a perception output from an autonomous vehicle as the autonomous vehicle traverses a segment of a route, the perception output being based on sensor data generated by a sensor suite of the autonomous vehicle; and
generating, for an output device of a user, a human-perceptible three-dimensional rendering of an external environment of the autonomous vehicle based on the perception output;
wherein generating the human-perceptible three-dimensional rendering of the external environment includes updating the human-perceptible three-dimensional rendering as the user performs driving operations for the autonomous vehicle to traverse the segment of the route.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein generating the human-perceptible three-dimensional rendering of the external environment includes generating, for each of multiple objects of multiple types that are identified by the perception output, a corresponding representation that reflects one or more physical characteristics of the type of object.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein the multiple objects include one or more objects that are in motion, and wherein the corresponding representation of each object of the one or more objects is generated to be dynamic to reflect a motion of the object.

### Claim 14 (depends on 13)

The computing system of claim 13, wherein the multiple objects include a pedestrian that is on or near a roadway on which the autonomous vehicle operates.

### Claim 15 (depends on 13)

The computing system of claim 13, wherein the multiple objects include a second vehicle that is adjacent to or within proximity of the autonomous vehicle.

### Claim 16 (depends on 11)

The computing system of claim 11, wherein generating the human-perceptible three-dimensional rendering of the external environment includes generating a representation of a traffic sign or signal that is identified by the perception output.

### Claim 17 (depends on 11)

The computing system of claim 11, wherein the human-perceptible three-dimensional representation is generated in real-time while the user performs the driving operations from within the autonomous vehicle.

### Claim 18 (depends on 17)

The computing system of claim 17, wherein the human-perceptible three-dimensional representation is rendered for a virtual reality headset worn by the user.

### Claim 19 (depends on 11)

The computing system of claim 11, wherein the human-perceptible three-dimensional representation is generated as a simulation for the user while the user performs the driving operations in a simulated environment.

### Claim 20

A computer-implemented method comprising:
receiving a perception output from an autonomous vehicle as the autonomous vehicle traverses a segment of a route, the perception output being based on sensor data generated by a sensor suite of the autonomous vehicle; and
generating, for an output device of a user, a human-perceptible three-dimensional rendering of an external environment of the vehicle based on the perception output;
wherein generating the human-perceptible three-dimensional rendering of the external environment includes updating the human-perceptible three-dimensional rendering as the user performs driving operations for the vehicle to traverse the segment of the route.
