# US 2026/0065688 A1 · Systems, Methods, and Apparatus for using Remote Assistance to Classify Objects in an Environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065688 A1
- Document ID: US20260065688A1
- Status: Published application
- Kind: A1
- Application number: 19/384,619
- Filed: 2025-11-10
- Published: 2026-03-05
- Earliest claimed: 2021-12-21
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Collin Winter; Vishay Nihalani
- Web page: https://patentgrep.com/patent/US20260065688A1

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)}
- G05D 1/2247: {providing the operator with simple or augmented images from one or more cameras}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G05D 2111/60: Combination of two or more signals
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

Example embodiments relate to techniques for enabling one or more systems of a vehicle (e.g., an autonomous vehicle) to request remote assistance to help the vehicle navigate in an environment. A computing device may be configured to receive a request for assistance from a vehicle to classify an object and to initiate display of a graphical user interface at a display device. The graphical user interface may be configured to visually represent the object and may comprise one or more graphical user interface elements to enable input to be provided for classifying the object. The computing device may also be configured to generate a response that includes a classification of the object based on detecting a selection of at least one of the one or more graphical user interface elements. Further, the computing device may be configured to transmit the response to the vehicle.

## Claims

### Claim 1

A method for providing remote assistance to classify an object, comprising:
receiving, at a remote computing device, a request from an autonomous vehicle to classify an object in an environment, the request including sensor data representative of the object and at least one classification determined by the vehicle;
causing display of a graphical user interface (GUI) at a display device, wherein the GUI is configured to:
display a representation of the object based on the sensor data;
display the at least one classification determined by the vehicle; and
provide one or more interactive GUI elements configured to enable an operator to select the at least one classification determined by the vehicle or input a new classification for the object;
detecting operator interaction with the one or more interactive GUI elements; and
transmitting, to the vehicle, a response including a final classification for the object for use in autonomous navigation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the GUI further displays a confidence score associated with the at least one classification determined by the vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein the request includes a plurality of classifications determined by the vehicle, and the GUI displays a listing of the plurality of classifications.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more interactive GUI elements include at least one of a dropdown menu, a radio button, or a text input field.

### Claim 5 (depends on 1)

The method of claim 1, wherein the GUI further displays contextual information about at least one of the vehicle or the environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein the response further includes a navigation instruction for the vehicle.

### Claim 7 (depends on 1)

The method of claim 1, wherein the vehicle is configured to pause navigation upon sending the request.

### Claim 8 (depends on 1)

The method of claim 1, wherein the final classification is used by the vehicle to update a control strategy.

### Claim 9 (depends on 1)

The method of claim 1, wherein the remote computing device communicates with the vehicle via a wireless network.

### Claim 10 (depends on 3)

The method of claim 3, wherein the plurality of classifications are determined by the vehicle using different sensor types.

### Claim 11 (depends on 3)

The method of claim 3, wherein the GUI displays confidence scores for the plurality of classifications, enabling the operator to compare confidence levels when making a selection.

### Claim 12 (depends on 3)

The method of claim 3, wherein the request is triggered by the vehicle determining different classifications for the object at different time periods.

### Claim 13 (depends on 1)

The method of claim 1, wherein the GUI is configured to enable the operator to provide a custom classification different from the at least one classification determined by the vehicle.

### Claim 14

A system for remote object classification, comprising:
a remote computing device configured to:
receive a request for assistance from an autonomous vehicle to classify an object, the request including sensor data and at least one vehicle-determined classification;
initiate display of a graphical user interface (GUI) at a display device, wherein the GUI is configured to:
display the sensor data;
display the at least one vehicle-determined classification; and
provide interactive elements enabling an operator to select the at least one vehicle-determined classification or input a different classification; and
generate and transmit a response to the vehicle including a final classification based on operator input.

### Claim 15 (depends on 14)

The system of claim 14, wherein the GUI displays a confidence score associated with the at least one vehicle-determined classification.

### Claim 16 (depends on 14)

The system of claim 14, wherein the sensor data includes image data and lidar data.

### Claim 17 (depends on 14)

The system of claim 14, wherein the GUI includes a visual representation of the object and a listing of multiple vehicle-determined classifications.

### Claim 18 (depends on 14)

The system of claim 14, wherein the vehicle is configured to use the final classification to adjust a navigation path.

### Claim 19 (depends on 17)

The system of claim 17, wherein the multiple vehicle-determined classifications include at least a first classification based on image data and a second classification based on lidar data.

### Claim 20

A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor of a remote computing device, cause the processor to perform operations comprising:
receiving a classification assistance request from an autonomous vehicle, the request including sensor data of an object and a plurality of vehicle-generated classification labels;
initiating presentation of a graphical user interface (GUI) to a human operator, wherein the GUI includes:
a visual representation of the object from the sensor data;
a listing of the plurality of vehicle-generated classification labels; and
interactive controls enabling the operator to:
accept one of the vehicle-generated classification labels, or
override the vehicle-generated classification labels with a different classification; and
in response to operator input via the interactive controls, sending a classification response to the vehicle to adjust a navigation strategy.
