# US 2021/0278837 A1 · Remote Assistance for Autonomous Vehicles in Predetermined Situations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278837 A1
- Document ID: US20210278837A1
- Status: Published application
- Kind: A1
- Application number: 17/329,020
- Filed: 2021-05-24
- Published: 2021-09-09
- Earliest claimed: 2014-03-03
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Nathaniel Fairfield; Joshua Selh Herbach; Vadim Furman
- Web page: https://patentgrep.com/patent/US20210278837A1

## CPC

- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- B60W 30/00: Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W 60/001: {Planning or execution of driving tasks}
- B60W 60/00253: {Taxi operations}
- G05D 1/0011: {associated with a remote control arrangement}
- G05D 1/0027: {involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00)}
- G05D 1/0033: {by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle}
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- B60W 2540/049: Number of occupants
- B60W 2556/45: External transmission of data to or from the vehicle
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Example systems and methods enable an autonomous vehicle to request assistance from a remote operator in certain predetermined situations. One example method includes determining a representation of an environment of an autonomous vehicle based on sensor data of the environment. Based on the representation, the method may also include identifying a situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance. The method may further include sending a request for assistance to a remote assistor, the request including the representation of the environment and the identified situation. The method may additionally include receiving a response from the remote assistor indicating an autonomous operation. The method may also include causing the autonomous vehicle to perform the autonomous operation.

## Claims

### Claim 1

A method comprising:
determining, by a first computing device, a route for a vehicle involves an unprotected turn, wherein the vehicle is navigating the route in an autonomous mode;
based on determining that the vehicle is approaching the unprotected turn, providing a video feed to a second computing device;
receiving, at the first computing device, instructions from the second computing device; and
controlling the vehicle based on the instructions.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
based on determining that the vehicle is approaching the unprotected turn, determining that a turn signal is unavailable for the vehicle to use when performing the unprotected turn.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving the video feed from a camera coupled to a front portion of the vehicle; and
determining that the vehicle is approaching the unprotected turn based on the video feed.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
based on determining that the vehicle is approaching the unprotected turn, determining that a confidence level for performing the unprotected turn is below a threshold confidence level; and
responsive to determining that the confidence level for performing the unprotected turn is below the threshold confidence level, providing the video feed to the second computing device.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
responsive to determining that the confidence level for performing the unprotected turn is below the threshold confidence level, causing the vehicle to hold position until receiving a response from the second computing device.

### Claim 6 (depends on 1)

The method of claim 1, wherein providing the video feed to the second computing device comprises:
transmitting the video feed to a remote computing device positioned remotely from the vehicle.

### Claim 7 (depends on 1)

The method of claim 1, wherein providing the video feed to the second computing device comprises:
transmitting the video feed to a computing device associated with a passenger located in the vehicle.

### Claim 8 (depends on 1)

The method of claim 1, wherein providing the video feed to the second computing device comprises:
transmitting a request for assistance that includes the video feed and an indication of the route for the vehicle.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
based on determining the route for the vehicle involves the unprotected turn, transmitting, to the second computing device, an advanced notification indicating when the vehicle is estimated to approach the unprotected turn.

### Claim 10 (depends on 1)

The method of claim 1, wherein providing the video feed to the second computing device comprises:
transmitting the video feed to the second computing device with a latency indication, wherein the latency indication specifies a delay associated with the video feed.

### Claim 11 (depends on 1)

The method of claim 1, wherein controlling the vehicle based on the instructions comprises:
causing the vehicle to perform the unprotected turn.

### Claim 12

A system comprising:
a vehicle;
a first computing device coupled to the vehicle, wherein the first computing device is configured to:
determine a route for the vehicle involves an unprotected turn, wherein the vehicle is navigating the route in an autonomous mode;
based on determining that the vehicle is approaching the unprotected turn, provide a video feed to a second computing device;
receive instructions from the second computing device; and
control the vehicle based on the instructions.

### Claim 13 (depends on 12)

The system of claim 12, wherein the first computing device is further configured to:
determine that a turn signal is unavailable for the vehicle to use when performing the unprotected turn based on determining that the vehicle is approaching the unprotected turn.

### Claim 14 (depends on 12)

The system of claim 12, wherein the first computing device is further configured to:
receive the video feed from a camera coupled to a front portion of the vehicle; and
determine that the vehicle is approaching the unprotected turn based on the video feed.

### Claim 15 (depends on 12)

The system of claim 12, wherein the second computing device is a remote computing device positioned remotely from the vehicle.

### Claim 16 (depends on 12)

The system of claim 12, wherein the second computing device is associated with a passenger located in the vehicle.

### Claim 17 (depends on 12)

The system of claim 12, wherein the first computing device is further configured to:
transmit a request for assistance that includes the video feed, wherein the request for assistance includes an indication of the route for the vehicle.

### Claim 18 (depends on 12)

The system of claim 12, wherein the first computing device is further configured to:
transmit an advanced notification to the second computing device based on determining that the route for the vehicle involves the unprotected turn, wherein the advanced notification includes an indication of when the vehicle is estimated to approach the unprotected turn.

### Claim 19 (depends on 12)

The system of claim 12, wherein the first computing device is further configured to:
determine a confidence level for performing the unprotected turn is below a threshold confidence level; and
responsive to determining that the confidence level for performing the unprotected turn is below the threshold confidence level, cause the vehicle to hold position until receiving a response from the second computing device.

### Claim 20

A non-transitory computer readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:
determining a route for a vehicle involves an unprotected turn, wherein the vehicle is navigating the route in an autonomous mode;
based on determining that the vehicle is approaching the unprotected turn, providing a video feed to a computing device;
receiving navigation instructions from the computing device; and
controlling the vehicle based on the navigation instructions.
