# US 2020/0402323 A1 · Intervention in operation of a vehicle having autonomous driving capabilities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402323 A1
- Document ID: US20200402323A1
- Status: Published application
- Kind: A1
- Application number: 16/989,576
- Filed: 2020-08-10
- Published: 2020-12-24
- Earliest claimed: 2017-06-16
- Assignees: nuTonomy Inc.
- Applicants: nuTonomy Inc.
- Inventors: Shih-Yuan Liu; Harshavardhan Ravichandran; Karl Iagnemma; Hsun-Hsien Chang
- Web page: https://patentgrep.com/patent/US20200402323A1

## CPC

- G07C 5/006: {Indicating maintenance}
- B60W 20/50: Control strategies for responding to system failures, e.g. for fault diagnosis, failsafe operation or limp mode
- B60W 60/00186: {related to the vehicle}
- G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)}
- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- G05D 1/005: {by providing the operator with signals other than visual, e.g. acoustic, haptic}
- G05D 1/224: Output arrangements on the remote controller, e.g. displays, haptics or speakers
- G05D 1/225: operated by off-board computers
- G05D 1/226: Communication links with the remote-control arrangements
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)}
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Among other things, a determination is made that intervention in an operation of one or more autonomous driving capabilities of a vehicle is appropriate. Based on the determination, a person is enabled to provide information for an intervention. The intervention is caused in the operation of the one or more autonomous driving capabilities of the vehicle.

## Claims

### Claim 1

A system comprising:
one or more processors; and
a non-transitory computer-readable storage medium storing instructions which when executed by the one or more processors cause the one or more processors to:
receive a request for an intervention from a vehicle;
treat a current location of the vehicle as a non-deterministic location having a conditional probability;
identify a geolocation of the vehicle based on probabilistic reasoning using the conditional probability; and
intervene in an operation of an autonomous driving capability of the vehicle.

### Claim 2 (depends on 1)

The system of claim 1, wherein the instructions that cause the one or more processors to receive the request cause the one or more processors to:
receive information about a status or environment of the vehicle or a related AV system, the status or the environment of the vehicle comprising a functionality of a hardware component or software of the vehicle or the AV system.

### Claim 3 (depends on 1)

The system of claim 1, wherein the instructions further cause the one or more processors to analyze the request to detect presence of unexpected data or absence of expected data, and wherein the instructions that cause the one or more processors to analyze the request cause the one or more processors to:
evaluate a mismatch between a measured quantity and a model-estimated quantity of a hardware component or software of the vehicle; or
use pattern recognition to evaluate an abnormal pattern in the request.

### Claim 4 (depends on 3)

The system of claim 3, wherein analyzing the request further comprises at least one of:
inferring a malfunction in the hardware component or the software;
detecting an unknown object present in an environment of the vehicle or a related AV system; or
inferring an event that is or will be happening in the environment of the vehicle or a related AV system.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions further cause the one or more processors to implement a fallback intervention in the operation of the at least one autonomous driving capability of the vehicle, and wherein the fallback intervention comprises at least one of:
causing the vehicle or a related AV system to enter a fully autonomous driving mode, a semi-autonomous driving mode, or a fully manual driving mode; or
causing the vehicle to operate at a reduced velocity.

### Claim 6 (depends on 5)

The system of claim 5, wherein the fallback intervention comprises at least one of:
identifying a safe-to-stop location;
generating a new trajectory to the safe-to-stop location;
invoking a backup hardware component or a backup software process; or
evaluating a functional hardware component or a functional software process required to operate the vehicle.

### Claim 7 (depends on 1)

The system of claim 1, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises at least one of:
evaluating one or more active events associated with the vehicle or a related AV system, or associated with the environment of the vehicle or a related AV system;
presenting a field of view or a bird's-eye of a vision sensor of the vehicle; or
presenting an interactive interface including presenting current or past or both trajectories.

### Claim 8 (depends on 1)

The system of claim 1, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises at least one of:
treating the current location as prior knowledge and using an inference algorithm to identify the geolocation;
inferring one or more trajectory segments based on one or more trajectory primitives; or
concatenating the trajectory segments by smoothing speed profiles across the trajectory segments.

### Claim 9 (depends on 1)

The system of claim 1, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises at least one of:
specifying one or more un-traversable road segments;
inferring a steering angle by a learning algorithm; or
enabling, editing or disabling a subcomponent of a hardware component or a processing step of a software process.

### Claim 10 (depends on 1)

The system of claim 1, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises editing data comprising at least one of a map, sensor data in the vehicle or a related AV system, trajectory data in the vehicle or a related AV system, vision data in the vehicle or a related AV system, or any past data in the vehicle or a related AV system.

### Claim 11

A non-transitory computer-readable storage medium storing instructions which when executed by one or more processors cause the one or more processors to:
receive a request for an intervention from a vehicle;
treat a current location of the vehicle as a non-deterministic location having a conditional probability;
identify a geolocation of the vehicle based on probabilistic reasoning using the conditional probability; and
intervene in an operation of an autonomous driving capability of the vehicle.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein receiving the request comprises receiving information about a status or environment of the vehicle or a related AV system, the status or the environment of the vehicle comprising a functionality of a hardware component or software of the vehicle or the AV system.

### Claim 13 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the instructions further cause the one or more processors to analyze the request to detect presence of unexpected data or absence of expected data, and wherein analyzing the request comprises at least one of:
evaluating a mismatch between a measured quantity and a model-estimated quantity of a hardware component or software of the vehicle; or
using pattern recognition to evaluate an abnormal pattern in the request.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein analyzing the request further comprises at least one of:
inferring a malfunction in the hardware component or the software;
detecting an unknown object present in an environment of the vehicle or a related AV system; or
inferring an event that is or will be happening in the environment of the vehicle or a related AV system.

### Claim 15 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the instructions further cause the one or more processors to implement a fallback intervention in the operation of the at least one autonomous driving capability of the vehicle, and wherein the fallback intervention comprises at least one of:
causing the vehicle or a related AV system to enter a fully autonomous driving mode, a semi-autonomous driving mode, or a fully manual driving mode; or
causing the vehicle to operate at a reduced velocity.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the fallback intervention comprises at least one of:
identifying a safe-to-stop location;
generating a new trajectory to the safe-to-stop location;
invoking a backup hardware component or a backup software process; or
evaluating a functional hardware component or a functional software process required to operate the vehicle.

### Claim 17 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises at least one of:
evaluating one or more active events associated with the vehicle or a related AV system, or associated with the environment of the vehicle or a related AV system;
presenting a field of view or a bird's-eye of a vision sensor of the vehicle; or
presenting an interactive interface including presenting current or past or both trajectories.

### Claim 18 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises at least one of:
treating the current location as prior knowledge and using an inference algorithm to identify the geolocation;
inferring one or more trajectory segments based on one or more trajectory primitives; or
concatenating the trajectory segments by smoothing speed profiles across the trajectory segments.

### Claim 19 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises at least one of:
specifying one or more un-traversable road segments;
inferring a steering angle by a learning algorithm; or
enabling, editing or disabling a subcomponent of a hardware component or a processing step of a software process.

### Claim 20 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein intervening in the operation of the autonomous driving capability of the vehicle comprises editing data comprising at least one of a map, sensor data in the vehicle or a related AV system, trajectory data in the vehicle or a related AV system, vision data in the vehicle or a related AV system, or any past data in the vehicle or a related AV system.
