# US 2021/0004012 A1 · Goal-Directed Occupancy Prediction for Autonomous Driving

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0004012 A1
- Document ID: US20210004012A1
- Status: Published application
- Kind: A1
- Application number: 16/853,151
- Filed: 2020-04-20
- Published: 2021-01-07
- Earliest claimed: 2019-07-05
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Micol Marchetti-Bowick; Poornima Kaniarasu; Galen Clark Haynes
- Web page: https://patentgrep.com/patent/US20210004012A1

## CPC

- B60W 30/18154: {Approaching an intersection}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- B60W 30/18163: {Lane change; Overtaking manoeuvres}
- B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles}
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G01C 21/3407: {specially adapted for specific applications}
- G01C 21/3446: {Details of route searching algorithms, e.g. Dijkstra, A*, arc-flags or using precalculated routes}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0221: {involving a learning process}
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

An autonomous vehicle can obtain state data associated with an object in an environment, obtain map data including information associated with spatial relationships between at least a subset of lanes of a road network, and determine a set of candidate paths that the object may follow in the environment based at least in part on the spatial relationships between at least two lanes of the road network. Each candidate path can include a respective set of spatial cells. The autonomous vehicle can determine, for each candidate path, a predicted occupancy for each spatial cell of the respective set of spatial cells of such candidate path during at least a portion of a prediction time horizon. The autonomous vehicle can generate prediction data associated with the object based at least in part on the predicted occupancy for each spatial cell of the respective set of spatial cells for at least one candidate path.

## Claims

### Claim 1

A computer-implemented method of object prediction for autonomous driving, the method comprising:
obtaining, by a computing system comprising one or more computing devices and based at least in part on sensor data associated with a set of sensors of an autonomous vehicle, state data associated with an object in an environment external to the autonomous vehicle;
obtaining, by the computing system, map data including information associated with spatial relationships between at least a subset of lanes of a road network associated with the environment;
determining, by the computing system, a set of candidate paths that the object may follow in the environment based at least in part on the spatial relationships between at least two lanes of the road network, wherein each candidate path includes a respective set of spatial cells;
determining, by the computing system for each candidate path of the set of candidate paths, a predicted occupancy for each spatial cell of the respective set of spatial cells of such candidate path during at least a portion of a prediction time horizon; and
generating, by the computing system, prediction data associated with the object based at least in part on the predicted occupancy for each spatial cell of the respective set of spatial cells for at least one candidate path.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
discretizing each candidate path of the set of candidate paths into the respective set of spatial cells of such candidate path.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein determining the predicted occupancy for each spatial cell of the respective set of a spatial cells for at least one candidate path comprises:
determining the predicted occupancy for such spatial cell for a duration of the prediction time horizon.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein determining the predicted occupancy for each spatial cell of the respective set of spatial cells for at least one candidate path comprises:
determining a plurality of predicted occupancies for such spatial cell for each of a plurality of time periods over of the prediction time horizon.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a lane graph comprising a plurality of nodes associated with a plurality of lane segments of a plurality of lanes, the lane graph comprising a plurality of edges that represent connections between lane segments.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein determining, by the computing system for each candidate path of the set of predicted paths, the predicted occupancy for each spatial cell of the respective set of spatial cells of such candidate path, comprises:
jointly predicting occupancy over the lane graph for a plurality of objects.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein generating the lane graph comprises:
converting the respective set of cells for each candidate path of the set of candidate paths into nodes in the lane graph with edges connecting the nodes.

### Claim 8 (depends on 5)

The computer-implemented method of claim 5, wherein determining, by the computing system for each candidate path of the set of candidate paths, the predicted occupancy for each spatial cell of the respective set of spatial cells of such candidate path, comprises:
providing data associated with a subset of the nodes corresponding to such candidate path as input to one or more machine-learned occupancy models.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein:
the map data includes an overhead view of a rasterized map of at least a portion of the environment.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein:
the map data includes a respective raster for each lane segment of a plurality of lane segments for each lane of the subset of lanes of the network; and
the respective rasterization for at least one lane segment of at least one lane includes contextual information associated with such lane segment.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating, by the computing system, a motion plan for the autonomous vehicle based at least in part on the prediction data associated with the object.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein determining the predicted occupancy for each spatial cell of the respective set of spatial cells of each candidate path during at least the portion of the prediction time horizon comprises:
providing data indicative of such candidate path as input to one or more machine-learned occupancy models;
providing data associated with one or more scene features as input to the one or more machine-learned occupancy models;
providing state data associated with the actor as input to the one or more machine-learned occupancy models; and
generating, by the computing system using the one or more machine-learned occupancy models, data indicative of the predicted occupancy for such spatial cell.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the data associated with one or more scene features is generated by:
inputting rasterized map data to one or more feature extraction networks of the one or more machine-learned occupancy models and receiving as output of the one or more feature extraction networks the data associated with the one or more scene features.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the map data encodes a topology of a road network within the environment.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, further comprising:
selecting, by the computing system, a first candidate path from the set of candidate paths based on the predicted occupancies for the sets of spatial cells corresponding to the sets of candidate paths; and
wherein the prediction data associated with the object is generated based at least in part on the first candidate path.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein detecting, by the computing system and based at least in part on sensor data associated with the set of sensors of the autonomous vehicle, the object in the environment external to the autonomous vehicle comprises:
inputting the sensor data to one or more machine-learned models of a perception computing system of the autonomous vehicle; and
receiving as an output of the one or more machine-learned models data indicative of an object detection associated with the object.

### Claim 17

An autonomous vehicle, comprising:
a sensor system configured to generate sensor data of an environment external to the autonomous vehicle;
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned model configured to estimate occupancies for candidate paths that a detected object may follow in the environment based at least in part on spatial relationships between lanes of a road network associated with the environment; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
inputting, to the machine-learned model, state data associated with the object and map data including information associated with spatial relationships between at least a subset of lanes of the road network associated with the environment;
determining, by the machine-learned model in response to the state data and the map data, a candidate path that the object may follow in the environment, wherein the candidate path includes a set of spatial cells;
determining, by the machine-learned model, a predicted occupancy for each spatial cell of the set of spatial cells for the candidate path; and
generating prediction data associated with the object based at least in part on the predicted occupancy for each spatial cell of the set of spatial cells for the candidate path.

### Claim 18 (depends on 17)

The autonomous vehicle of claim 17, wherein the operations further comprise:
discretizing each predicted path of the set of predicted paths into the respective set of spatial cells of such predicted path.

### Claim 19

One or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
inputting, to a machine-learned model, state data associated with an object and map data including information associated with spatial relationships between at least a subset of lanes of a road network associated with an environment external to an autonomous vehicle;
determining, by the machine-learned model in response to the state data and the map data, a candidate path that the object may follow in the environment, wherein the candidate path includes a set of spatial cells;
determining, by the machine-learned model, a predicted occupancy for each spatial cell of the set of spatial cells for the candidate path; and
generating prediction data associated with the object based at least in part on the predicted occupancy for each spatial cell of the set of spatial cells for the candidate path.

### Claim 20 (depends on 19)

The one or more non-transitory computer-readable media of claim 19, wherein the operations further comprise:
generating a motion plan for the autonomous vehicle based at least in part on the prediction data associated with the object.
