# US 2024/0005576 A1 · Systems and methods for displaying autonomous vehicle environmental awareness

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005576 A1
- Document ID: US20240005576A1
- Status: Published application
- Kind: A1
- Application number: 18/347,476
- Filed: 2023-07-05
- Published: 2024-01-04
- Earliest claimed: 2018-09-14
- Assignees: Lyft, Inc.
- Applicants: Lyft, Inc.
- Inventors: Han Han Xue; Jisi Guo; Katherine Marisa Dill; Johan Ulrich Lewin Jessen; Audrey Liu; Oleg Vadim Panichev; Robert Earl Rasmusson
- Web page: https://patentgrep.com/patent/US20240005576A1

## CPC

- G06T 11/60: Creating or editing images; Combining images with text
- G06T 5/20: using local operators
- G06T 5/70: Denoising; Smoothing
- B60R 2300/60: characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR

## Abstract

The disclosed computer-implemented method may include displaying vehicle environment awareness. In some embodiments, a visualization system may display an abstract representation of a vehicle's physical environment via a mobile device and/or a device embedded in the vehicle. For example, the visualization may use a voxel grid to represent the environment and may alter characteristics of shapes in the grid to increase their visual prominence when the sensors of the vehicle detect that an object is occupying the space represented by the shapes. In some embodiments, the visualization may gradually increase and reduce the visual prominence of shapes in the grid to create a soothing wave effect. Various other methods, systems, and computer-readable media are also disclosed.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A computer-implemented method comprising:
providing a visualization comprising a representation of a vehicle and a representation of a physical environment associated with the vehicle using sensor data from at least one sensor of the vehicle that is configured to capture at least a portion of the physical environment, the visualization including an array of graphical elements that at least partially surrounds the representation of the vehicle;
determining a predicted path for the vehicle indicating where the vehicle is predicted to travel;
altering at least a portion of the visualization by dynamically adding an indicator of the predicted path to the visualization and by further updating the array of graphical elements based on the determined predicted path of the vehicle; and
providing an updated visualization showing the dynamically added predicted path indicator that was generated according to the sensor data and showing the updated array of graphical elements.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein the sensor data indicates at least one of the following: acceleration of the vehicle, deceleration of the vehicle, steering of the vehicle, a current location of the vehicle according to navigation system data, or one or more nearby objects that are within range of the at least one sensor.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, wherein the visualization includes one or more portions of additional data substantially next to the vehicle representation or the physical environment representation.

### Claim 24 (depends on 23)

The computer-implemented method of claim 23, wherein the additional data includes at least one of: a current speed of the vehicle, a current fuel economy of the vehicle, a distance traveled in a current trip, a remaining distance the current trip, or a distance to or from a transfer point.

### Claim 25 (depends on 21)

The computer-implemented method of claim 21, wherein the visualization includes one or more indicators representing traffic control devices encountered by the vehicle, including at least one of: a traffic light, a stop sign, a yield sign, or a turn sign.

### Claim 26 (depends on 21)

The computer-implemented method of claim 21, wherein one or more visual aesthetics of the predicted path indicator are varied based on a determined confidence level in the accuracy of the generated predicted path based on the sensor data received from the vehicle.

### Claim 27 (depends on 21)

The computer-implemented method of claim 21, wherein the predicted path indicator is presented as opaque when the vehicle is moving and is presented as transparent when the vehicle is stationary.

### Claim 28 (depends on 27)

The computer-implemented method of claim 27, wherein the predicted path indicator changes opacity according to a current speed of the vehicle.

### Claim 29 (depends on 21)

The computer-implemented method of claim 21, wherein upon determining that the vehicle is changing direction by at least a minimum specified amount, the predicted path indicator is dynamically changed in the visualization.

### Claim 30 (depends on 21)

The computer-implemented method of claim 21, wherein the predicted path indicator is generated according to data gathered from an autonomous vehicle path planner that, itself, is configured to plot a trajectory of the vehicle over a future time period.

### Claim 31 (depends on 21)

The computer-implemented method of claim 21, wherein the predicted path indicator is generated based on a route indicator that has access to information about at least one destination of the vehicle.

### Claim 32 (depends on 21)

The computer-implemented method of claim 21, wherein the vehicle is an autonomous vehicle, and wherein the visualization is changed to reflect that a human operator has taken control of the autonomous vehicle.

### Claim 33 (depends on 32)

The computer-implemented method of claim 32, further comprising ceasing to display the predicted path indicator upon determining that a human operator has taken control of the autonomous vehicle.

### Claim 34 (depends on 21)

The computer-implemented method of claim 21, wherein providing the visualization comprises:
sending the visualization to a mobile device allocated to the vehicle; and
displaying the visualization on a display device of the mobile device associated with the vehicle.

### Claim 35 (depends on 21)

The computer-implemented method of claim 21, wherein providing the visualization comprises sending the visualization to a mobile computing device associated with a passenger within the vehicle.

### Claim 36 (depends on 21)

The computer-implemented method of claim 21, wherein the representation of the physical environment associated with the vehicle begins a predetermined radius away from the representation of the vehicle.

### Claim 37

A system comprising:
a non-transitory memory; and
one or more hardware processors configured to execute instructions from the non-transitory memory to perform operations comprising:
providing a visualization comprising a representation of a vehicle and a representation of a physical environment associated with the vehicle using sensor data from at least one sensor of the vehicle that is configured to capture at least a portion of the physical environment, the visualization including an array of graphical elements that at least partially surrounds the representation of the vehicle;
determining a predicted path for the vehicle indicating where the vehicle is predicted to travel;
altering at least a portion of the visualization by dynamically adding an indicator of the predicted path to the visualization and by further updating the array of graphical elements based on the determined predicted path of the vehicle; and
providing an updated visualization showing the dynamically added predicted path indicator that was generated according to the sensor data and showing the updated array of graphical elements.

### Claim 38 (depends on 37)

The system of claim 37, wherein the array of graphical elements is altered to indicate the presence of one or more objects including at least one additional vehicle.

### Claim 39 (depends on 37)

The system of claim 37, wherein providing the visualization comprises:
sending the visualization to a mobile device allocated to the vehicle; and
displaying the visualization on a display device of the mobile device associated with the vehicle.

### Claim 40

A computer-readable medium comprising:
computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
provide a visualization comprising a representation of a vehicle and a representation of a physical environment associated with the vehicle using sensor data from at least one sensor of the vehicle that is configured to capture at least a portion of the physical environment, the visualization including an array of graphical elements that at least partially surrounds the representation of the vehicle;
determine a predicted path for the vehicle indicating where the vehicle is predicted to travel;
alter at least a portion of the visualization by dynamically adding an indicator of the predicted path to the visualization and by further updating the array of graphical elements based on the determined predicted path of the vehicle; and
provide an updated visualization showing the dynamically added predicted path indicator that was generated according to the sensor data and showing the updated array of graphical elements.
