# US 2021/0278852 A1 · Systems and Methods for Using Attention Masks to Improve Motion Planning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278852 A1
- Document ID: US20210278852A1
- Status: Published application
- Kind: A1
- Application number: 17/150,987
- Filed: 2021-01-15
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Raquel Urtasun; Bob Qingyuan Wei; Mengye Ren; Wenyuan Zeng; Ming Liang; Bin Yang
- Web page: https://patentgrep.com/patent/US20210278852A1

## CPC

- G05D 1/0217: {in accordance with energy consumption, time reduction or distance reduction criteria}
- B60W 60/001: {Planning or execution of driving tasks}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80)
- G05D 1/644: Optimisation of travel parameters, e.g. of energy consumption, journey time or distance
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 17/05: Geographic models
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for generating attention masks are provided. In particular, a computing system can access sensor data and map data for an area around an autonomous vehicle. The computing system can generate a voxel grid representation of the sensor data and map data. The computing system can generate an attention mask based on the voxel grid representation. The computing system can generate, by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map. The computing system can determine using the attention weighted feature map, a planning cost volume for an area around the autonomous vehicle. The computing system can select a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume.

## Claims

### Claim 1

A computer-implemented method for improving autonomous vehicle motion planning, the method comprising:
accessing, by a computing system including one or more processors, sensor data and map data for an area around an autonomous vehicle;
generating, by the computing system, a voxel grid representation of the sensor data and map data;
generating, by the computing system, an attention mask based on the voxel grid representation;
generating, by the computing system by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map;
determining, by the computing system, using the attention weighted feature map, a planning cost volume for an area around the autonomous vehicle; and
selecting, by the computing system, a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the attention mask comprises a grid of attention values, each respective attention value associated with a particular location in the voxel grid representation.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the attention value is a scalar value.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein a scalar value is associated with an importance associated with the particular location of the voxel grid representation.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein a scalar value is between 0 and 1 and generating, by the computing system using one or more machine-learned models, a feature map further comprising:
for each respective scalar value:
determining whether the respective scalar value exceeds a predetermined threshold value;
in accordance with a determination that the respective scalar value exceeds the predetermined threshold value, setting the respective scalar value to one; and
in accordance with a determination that the respective scalar value does not exceed the predetermined threshold value, setting the respective scalar value to zero.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein generating, by the computing system by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map further comprises:
generating, by the computing system, a sparse feature map by multiplying the voxel grid representation and the attention mask; and
receiving, by the computing system, the attention weighted feature map as output from one or more machine-learned models using the sparse feature map and the attention mask as input.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein generating, by the computing system by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map further comprises:
concatenating, by the computing system, the attention weighted feature map with the voxel grid representation.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the planning cost volume comprises a cost associated with each position in the planning cost volume.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the planning cost volume comprises costs associated with each position and time step described by the planning cost volume.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein selecting, by the computing system, a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume further comprises:
generating, by the computing system, a plurality of candidate trajectories for the autonomous vehicle, each candidate trajectory comprising a series of points that represent a route of the autonomous vehicle through the area around the autonomous vehicle, each point being associated with a particular point in time.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein selecting, by the computing system, a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume further comprises:
for a respective candidate trajectory in the plurality of candidate trajectories:
determining, by the computing system, a cost for each respective point in the series of points associated with the respective trajectory by using a location of the respective point and the associated point in time to identify a predicted cost of the point in the planning cost volume; and
aggregating, by the computing system, the costs for each position in the series of points into a total cost associated for the respective candidate trajectory.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein selecting, by the computing system, a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume further comprises:
selecting, by the computing system, the candidate trajectory with a lowest total cost.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, further comprises:
generating, by the computing system, a list of detected objects based, at least in part, on the attention weighted feature map.

### Claim 14

A computing system for improving autonomous vehicle motion planning for an autonomous vehicle, the system comprising:
one or more processors and one or more non-transitory computer-readable memories;
wherein the one or more non-transitory computer-readable memories store instructions that, when executed by the processor, cause the computing system to perform operations, the operations comprising:
accessing sensor data and map data for an area around the autonomous vehicle;
generating a voxel grid representation of the sensor data and map data;
generating an attention mask based on the voxel grid representation;
generating, by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map;
determining, by using the attention weighted feature map, a planning cost volume for an area around the autonomous vehicle; and
selecting a trajectory for the autonomous vehicle based on the planning cost volume.

### Claim 15

A computer-implemented method for training a model, comprising:
providing, by a computing system with one or more processors, a set of training sensor data as input to one or more machine-learned models, the machine-learned models comprising a set of weights;
receiving, by the computing system, as output from the one or more machine-learned models, perception data and a binary attention mask;
generating, by the computing system, perception loss data by determining a difference between the perception data received from the one or more machine-learned models and predetermined ground truth data associated with the training sensor data;
modifying, by the computing system, the perception loss data based on the binary attention mask; and
updating, by the computing system, the set of weights included in the one or more machine-learned models based on the modified perception loss data.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, further comprising:
receiving, by the computing system, as output from the one or more machine-learned models, a planning cost volume; and
determining, by the computing system, cost loss data based on the difference between the planning cost volume and a trajectory selected by a human operator in response to the set of training sensor data.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein updating, by the computing system, the set of weights included in the one or more machine-learned models comprises:
updating, by the computing system, the set of weights based on more two or more loss measurements comprising perception loss data and cost loss data.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, wherein the perception loss data is based on whether the received perception data comprises each object in the predetermined ground truth data and wherein modifying, by the computing system, the perception loss data based on binary attention mask further comprises:
determining, by the computing system for each object in the predetermined ground truth data, whether the object is within an attention area defined by the binary attention mask; and
down weighting, by the computing system, loss data associated with objects that are determined to be outside the attention area defined by the binary attention mask.

### Claim 19 (depends on 15)

The computer-implemented method of claim 15, wherein updating, by the computing system, the set of weights included in the one or more machine-learned models based on the modified perception loss data further comprises:
updating, by the computing system, the set of weights based on an attention size factor.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, wherein the attention size factor increases a determined loss as the size of the attention area in the binary attention mask updates.