# US 12,602,794 B2 · Method and unified framework system for full-stack autonomous driving planning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,794 B2
- Document ID: US12602794B2
- Status: Granted
- Kind: B2
- Application number: 18/306,516
- Filed: 2023-04-25
- Published: 2024-06-13
- Granted: 2026-04-14
- Earliest claimed: 2022-12-12
- Assignees: Shanghai Artificial Intelligence Innovation Center
- Applicants: Shanghai Artificial Intelligence Innovation Center
- Inventors: Hongyang Li; Li Chen; Jiazhi Yang; Yihan Hu; Chonghao Sima; Tianyu Li; Lewei Lu; Yu Liu; Qiang Liu; Junchi Yan; Dahua Lin; Yu Qiao; Xiaogang Wang
- Web page: https://patentgrep.com/patent/US12602794B2

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G06T 7/11: Region-based segmentation
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2556/40: High definition maps
- G01C 21/3811: {Point data, e.g. Point of Interest [POI]}
- G06T 2207/30241: Trajectory
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- Y02T 10/40: Engine management systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

The present application provides a method and a unified framework system for full-stack autonomous driving planning. The method comprises: acquiring an image of a scene and converting the image into an image feature, and converting the image feature into a bird's-eye view feature map; detecting agents from the bird's-eye view feature map through track query vectors, and continuously tracking the agents; segmenting different types of map elements from the bird's-eye view feature map through map query vectors, and continuously updating the map elements; predicting a future trajectory of each agent using an interaction between the agents and the different map elements; predicting, according to the predicted future trajectory of each agent, an occupancy grid map over multi-steps into the future; and decoding an ego-vehicle query vector to generate a planned path of an ego-vehicle, and optimizing the planned path using the predicted future multi-step occupancy grid map.

## Claims

### Claim 1

A full-stack autonomous driving planning method, comprising the following steps:
acquiring an image of a scene through a photographing unit, converting the image into an image feature through an image feature extractor, and converting the image feature into a bird's-eye view feature map through a bird's-eye view encoder;
detecting a plurality of agents from the bird's-eye view feature map through a group of track query vectors, and continuously tracking the plurality of agents, wherein the group of track query vectors are introduced to model an entire life cycle of the plurality of agents in a scene, wherein the entire life cycle is a whole process from appearance to complete disappearance;
segmenting different types of map elements from the bird's-eye view feature map through a group of map query vectors, and continuously updating the map elements;
predicting a future trajectory of each agent using an interaction between the plurality of agents and the different map elements, wherein the future trajectory of each agent is predicted by modeling the interaction between the plurality of agents and environment;
updating, according to the predicted future trajectory of each agent, a future bird's-eye view feature map through a multi-head attention mechanism which introduces motion information of objects by taking the bird's-eye view feature map as query vectors so as to predict an occupancy grid map over multi-steps into the future, wherein the occupancy grid map refers to a representation of a discretized bird's-eye view feature map with a value of each occupancy grid representing whether the occupancy grid is occupied by the agent; and
decoding an ego-vehicle query vector to generate a planned path of an ego-vehicle, and optimizing the planned path of the ego-vehicle by using the predicted future multi-step occupancy grid map, wherein the ego-vehicle query vector represents a modeling of an ego-vehicle motion state.

### Claim 2 (depends on 1)

The full-stack autonomous driving planning method according to claim 1, wherein keys and values of the track query vectors are the bird's-eye view feature map;
continuously tracking the plurality of agents comprises: updating the key and the value of each track query vector through a multi-head cross-attention mechanism.

### Claim 3 (depends on 1)

The full-stack autonomous driving planning method according to claim 1, wherein keys and values of the map query vectors are the bird's-eye view feature map;
continuously updating the map elements comprises: updating the key and the value of each map query vector through a multi-head cross-attention mechanism.

### Claim 4 (depends on 1)

The full-stack autonomous driving planning method according to claim 1, wherein the map elements comprise: lane lines and pedestrian lines.

### Claim 5

A unified framework system for full-stack autonomous driving planning, comprising:
an acquisition module, configured for acquiring an image of a scene, converting the image into an image feature through an image feature extractor, and converting the image feature into a bird's-eye view feature map through a bird's-eye view encoder;
a tracking module, configured for detecting a plurality of agents from the bird's-eye view feature map through a group of track query vectors, and continuously tracking the plurality of agents, wherein the group of track query vectors are introduced to model an entire life cycle of the plurality of agents in a scene, wherein the entire life cycle is a whole process from appearance to complete disappearance;
a map module, configured for segmenting different types of map elements from the bird's-eye view feature map through a group of map query vectors, and continuously updating the map elements;
a motion forecasting module, configured for forecasting a future trajectory of each agent using an interaction between the plurality of agents and the different map elements, wherein the future trajectory of each agent is predicted by modeling the interaction between the plurality of agents and environment;
an occupancy prediction module, configured for updating, according to the predicted future trajectory of each agent, a future bird's-eye view feature map through a multi-head attention mechanism which introduces motion information of objects by taking the bird's-eye view feature map as query vectors so as to predict an occupancy grid map over multi-steps into the future, wherein the occupancy grid map refers to a representation of a discretized bird's-eye view feature map with a value of each occupancy grid representing whether the occupancy grid is occupied by the agent; and
a planning module, configured for decoding an ego-vehicle query vector to generate a planned path of an ego-vehicle, and optimizing the planned path of the ego-vehicle by using the predicted future multi-step occupancy grid map, wherein the ego-vehicle query vector represents a modeling of an ego-vehicle motion state.

### Claim 6 (depends on 5)

The unified framework system according to claim 5, wherein keys and values of the track query vectors are the bird's-eye view feature map;
continuously tracking the plurality of agents comprises: updating the key and the value of each track query vector through a multi-head cross-attention mechanism.

### Claim 7 (depends on 5)

The unified framework system according to claim 5, wherein keys and values of the map query vectors are the bird's-eye view feature map;
continuously updating the map elements comprises: updating the key and the value of each map query vector through a multi-head cross-attention mechanism.

### Claim 8 (depends on 5)

The unified framework system according to claim 5, wherein the map elements comprise: lane lines and pedestrian lines.

### Claim 9 (depends on 1)

The full-stack autonomous driving planning method according to claim 1, wherein the interaction comprises interactions present between an agent and an agent, an agent and a map, and an agent and a trajectory end point respectively, and are represented as follows:
Q
a
/
m
=
MHCA
⁡
(
MHSA
⁡
(
Q
)
,
Q
A
/
Q
M
)
,
Q
g
=
DeformAttn
⁡
(
Q
,
x
^
T
l
-
1
,
B
)
,
where MHCA represents multi-head cross-attention, MHSA represents multi-head self-attention, DeformAttn represents deformable attention, Q represents a trajectory query vector, QA represents an agent feature from the tracking module, QM represents a map feature from the map module, Qa/m represents a trajectory feature after the interaction between track query vectors Q, QA and QM, B represents a bird's-eye view feature map, {circumflex over (x)}Tl-1 represents an output of the previous layer of motion forecasting module, and Qg represents a trajectory feature after the interaction between the trajectory query vector Q and the trajectory end point.

### Claim 10 (depends on 1)

The full-stack autonomous driving planning method according to claim 1, wherein an introduction of motion information of objects by multi-head attention mechanism comprises representing dense grids in the scene as query vectors Q, and representing object features as keys K and values V, wherein the query vectors are updated multiple times through multi-layer decoders and are defined to represent a future timing.