Method and unified framework system for full-stack autonomous driving planning

A unified framework system for full-stack autonomous driving planning, comprising:

• an acquisition module, configured for acquiring an image of a scene, converting the image into an image feature through an image feature extractor, and converting the image feature into a bird's-eye view feature map through a bird's-eye view encoder;
• a tracking module, configured for detecting a plurality of agents from the bird's-eye view feature map through a group of track query vectors, and continuously tracking the plurality of agents;
• a map module, configured for segmenting different types of map elements from the bird's-eye view feature map through a group of map query vectors, and continuously updating the map elements;
• a motion forecasting module, configured for forecasting a future trajectory of each agent using an interaction between the plurality of agents and the different map elements;
• an occupancy prediction module, configured for updating, according to the predicted future trajectory of each agent, a future bird's-eye view feature map through a multi-head attention mechanism by taking the bird's-eye view feature map as query vectors so as to predict an occupancy grid map over multi-steps into the future, wherein the occupancy grid map refers to a representation of a discretized bird's-eye view feature map with a value of each occupancy grid representing whether the occupancy grid is occupied by the agent;
• and
• a planning module, configured for decoding an ego-vehicle query vector to generate a planned path of an ego-vehicle, and optimizing the planned path of the ego-vehicle by using the predicted future multi-step occupancy grid map, wherein the ego-vehicle query vector represents a modeling of an ego-vehicle motion state.