# US 11,999,372 B2 · Operation of an autonomous vehicle based on availability of navigational information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,999,372 B2
- Document ID: US11999372B2
- Status: Granted
- Kind: B2
- Application number: 17/008,311
- Filed: 2020-08-31
- Published: 2021-03-18
- Granted: 2024-06-04
- Earliest claimed: 2019-09-12
- Assignees: Motional AD LLC
- Applicants: Motional Ad LLC
- Inventors: Jong Ho Lee; Rahul Ahluwalia; Jonathan Wieskamp
- Web page: https://patentgrep.com/patent/US11999372B2

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- G01S 19/485: {whereby the further system is an optical system or imaging system}
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- B60W 60/0053: {from vehicle to occupant}
- G01C 21/3407: {specially adapted for specific applications}
- G01C 21/3492: {employing speed data or traffic data, e.g. real-time or historical (traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968)}
- G01C 21/3833: {characterised by the source of data}
- G01C 21/3837: {Data obtained from a single source}
- G05D 1/0011: {associated with a remote control arrangement}
- G05D 1/223: Command input arrangements on the remote controller, e.g. joysticks or touch screens
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2555/00: Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
- B60W 2556/40: High definition maps
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- G01S 19/42: Determining position
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The subject matter described in this specification is directed to a computer system and techniques for operating an autonomous vehicle (AV) based on the availability of navigation information. In one example, while the AV is in an autonomous mode, first data is obtained from a first sensor and second data is obtained from a second sensor. A location of the AV is determined based on the first data and the second data. If navigational information is not available for the determined location, an alert is provided that the AV will exit the autonomous mode.

## Claims

### Claim 1

A system, comprising:
one or more computer processors; and
one or more non-transitory storage media storing instructions which, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
operating a vehicle in an autonomous mode, wherein operating the vehicle in the autonomous mode comprises:
receiving sensor data associated with one or more objects within an environment of the vehicle;
causing control of at least one of steering, throttling or ignition of the vehicle based on the sensor data to navigate the vehicle to a destination;
determining a first geographic location of the vehicle using location data from a location sensor;
determining a second geographic location of the vehicle using the first geographic location, imaging data from an imaging sensor, and a semantic map comprising a map with annotated features, wherein the second geographic location is more precise than the first geographic location;
determining a predicted location of the vehicle based on the second geographic location of the vehicle;
in accordance with a determination that the annotated features of the semantic map are not available for determining geographical locations at the predicted location, providing an alert that the vehicle will exit the autonomous mode; and
in accordance with a determination that no operator control is detected after providing the alert:
maintaining operation of the vehicle in the autonomous mode; and
re-routing the vehicle by navigating the vehicle on a route that keeps the vehicle within an area in which the annotated features of the semantic map are available for determining geographical locations.

### Claim 2 (depends on 1)

The system of claim 1, wherein the instructions further cause the one or more computer processors to perform operations comprising:
in accordance with a determination that the annotated features of the semantic map are available for the predicted location, continuing operating the vehicle in the autonomous mode.

### Claim 3 (depends on 1)

The system of claim 1, wherein the instructions further cause the one or more computer processors to perform operations comprising:
in accordance with a determination that operator control is detected after providing the alert:
ceasing the autonomous mode; and
initiating an operator assisted mode.

### Claim 4 (depends on 3)

The system of claim 3, wherein the instructions further cause the one or more computer processors to perform operations comprising:
while the vehicle is operating in the operator assisted mode:
receiving one or more commands for navigating the vehicle from a navigation system remote from the vehicle; and
navigating the vehicle according to the one or more commands.

### Claim 5 (depends on 3)

The system of claim 3, wherein the instructions further cause the one or more computer processors to perform operations comprising:
while the vehicle is operating in the operator assisted mode:
capturing one or more features of the environment; and
providing the one or more captured features to a server to update the semantic map.

### Claim 6 (depends on 3)

The system of claim 3, wherein the instructions further cause the one or more computer processors to perform operations comprising:
while the vehicle is operating in the operator assisted mode:
obtaining first data from the location sensor;
obtaining second data from the imaging sensor;
determining a predicted third geographic location based on the first data obtained from the location sensor and the second data obtained from the imaging sensor;
in accordance with a determination that the annotated features of the semantic map are not available for determining geographical locations at the predicted third geographic location:
maintaining the operator assisted mode; and
in accordance with a determination that the annotated features of the semantic map are available for determining geographical locations at the predicted third geographic location:
ceasing the operator assisted mode; and
initiating the autonomous mode.

### Claim 7 (depends on 1)

The system of claim 1, wherein the predicted location of the vehicle based on the second geographic location of the vehicle is a predicted future location of the vehicle.

### Claim 8 (depends on 1)

The system of claim 1, wherein the instructions further cause the one or more computer processors to perform operations comprising:
determining that the annotated features of the semantic map are not available for determining geographical locations based on a determination that the imaging data from the imaging sensor or the location data from the location sensor is unavailable or inaccurate, wherein the location sensor is a GPS sensor.

### Claim 9 (depends on 1)

The system of claim 1, wherein the instructions further cause the one or more computer processors to perform operations comprising:
determining that the annotated features of the semantic map are not available for determining geographical locations based on a determination that the imaging data from the imaging sensor does not match data from the semantic map.

### Claim 10

A method, comprising:
operating a vehicle in an autonomous mode, wherein operating the vehicle in the autonomous mode comprises:
receiving sensor data associated with one or more objects within an environment of the vehicle; and
causing control of at least one of steering, throttling or ignition of the vehicle based on the sensor data to navigate the vehicle to a destination;
determining a first geographic location of the vehicle using location data from a location sensor;
determining a second geographic location of the vehicle using the first geographic location, imaging data from an imaging sensor, and a semantic map comprising a map with annotated features, wherein the second geographic location is more precise than the first geographic location;
determining a predicted location of the vehicle based on the second geographic location of the vehicle;
in accordance with a determination that the annotated features of the semantic map are not available for determining geographical locations at the predicted location, providing an alert that the vehicle will exit the autonomous mode; and
in accordance with a determination that no operator control is detected after providing the alert:
maintaining operation of the vehicle in the autonomous mode; and
re-routing the vehicle by navigating the vehicle on a route that keeps the vehicle within an area in which the annotated features of the semantic map are available for determining geographical locations.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
in accordance with a determination that the annotated features of the semantic map are available for the predicted location, continuing operating the vehicle in the autonomous mode.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
in accordance with the determination that the annotated features of the semantic map are not available for determining geographical locations at the predicted location, navigating the vehicle without the imaging data from the imaging sensor.

### Claim 13 (depends on 10)

The method of claim 10, wherein the annotated features of the semantic map are not available for determining geographical locations at the predicted location when the predicted location corresponds to a region without the semantic map.

### Claim 14 (depends on 10)

The method of claim 10, further comprising:
in accordance with a determination that operator control is detected after providing the alert:
ceasing the autonomous mode; and
initiating an operator assisted mode.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
while the vehicle is operating in the operator assisted mode:
receiving one or more commands for navigating the vehicle from a navigation system remote from the vehicle; and
navigating the vehicle according to the one or more commands.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
while the vehicle is operating in the operator assisted mode:
capturing one or more features of the environment; and
providing the one or more captured features to a server to update the semantic map.

### Claim 17

One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
operating a vehicle in an autonomous mode, wherein operating the vehicle in the autonomous mode comprises:
receiving sensor data associated with one or more objects within an environment of the vehicle; and
causing control of at least one of steering, throttling or ignition of the vehicle based on the sensor data to navigate the vehicle to a destination;
determining a first geographic location of the vehicle using location data from a location sensor;
determining a second geographic location of the vehicle using the first geographic location, imaging data from an imaging sensor, and a semantic map comprising a map with annotated features, wherein the second geographic location is more precise than the first geographic location;
determining a predicted location of the vehicle based on the second geographic location of the vehicle;
in accordance with a determination that the annotated features of the semantic map are not available for determining geographical locations at the predicted location, providing an alert that the vehicle will exit the autonomous mode; and
in accordance with a determination that no operator control is detected after providing the alert:
maintaining operation of the vehicle in the autonomous mode; and
re-routing the vehicle by navigating the vehicle on a route that keeps the vehicle within an area in which the annotated features of the semantic map are available for determining geographical locations.