# US 12,008,454 B2 · Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,454 B2
- Document ID: US12008454B2
- Status: Granted
- Kind: B2
- Application number: 18/186,718
- Filed: 2023-03-20
- Published: 2023-07-20
- Granted: 2024-06-11
- Earliest claimed: 2019-07-08
- Assignees: Uatc, LLC
- Applicants: Uatc, LLC
- Inventors: Raquel Urtasun; Renjie Liao; Sergio Casas; Cole Christian Gulino
- Web page: https://patentgrep.com/patent/US12008454B2

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]}
- G01C 21/3626: {Details of the output of route guidance instructions (traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968)}
- G01C 21/3811: {Point data, e.g. Point of Interest [POI]}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/06: Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 7/046: {Implementation by means of a neural network (neural networks using fuzzy logic G06N3/043)}
- G08G 1/0133: {for classifying traffic situation}
- G08G 1/0141: {for traffic information dissemination}
- G08G 1/096775: {where the origin of the information is a central station}
- G08G 1/20: {Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same are disclosed. The computing system can include an object detection model and a graph neural network including a plurality of nodes and a plurality of edges. The computing system can be configured to input sensor data into the object detection model; receive object detection data describing the location of the plurality of the actors relative to the autonomous vehicle as an output of the object detection model; input the object detection data into the graph neural network; iteratively update a plurality of node states respectively associated with the plurality of nodes; and receive, as an output of the graph neural network, the motion forecast data with respect to the plurality of actors.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, from one or more sensors, sensor data associated with an environment of an autonomous vehicle;
generating, using an object detection model, object detection data based on the sensor data, the object detection data describing a location of an actor relative to the autonomous vehicle;
generating, using a graph neural network, motion forecast data associated with the actor, wherein the graph neural network comprises a plurality of nodes and a plurality of edges, the plurality of nodes indicative of a plurality of actors including the actor and the edges indicative of relationships among the plurality of actors,
wherein the graph neural network is configured to update at least one node state of a first node by passing at least one message between at least two nodes comprising the first node; and
determining a motion plan for the autonomous vehicle based on the motion forecast data; and
controlling the autonomous vehicle based on the motion plan.

### Claim 2 (depends on 1)

The computer-implemented method claim 1, comprising:
obtaining map data associated with the environment of the autonomous vehicle;
concatenating, using the object detection model, the map data and the sensor data; and
generating, using the object detection model, the object detection data based on the map data and the sensor data.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the graph neural network is indicative of at least one of: (i) a partially connected graph neural network, or (ii) a fully connected graph neural network.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one node state is indicative of at least one of: (i) a future trajectory of a respective actor of the plurality of actors or (ii) a behavior of the respective actor.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein passing the at least one message between the at least two nodes is indicative of a potential interaction among the actors respectively associated with the at least two nodes.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the at least one message is indicative of a spatial relationship.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the sensor data comprises three-dimensional LiDAR data.

### Claim 8

A vehicle computing system comprising:
one or more processors; and
one or more memory resources storing instructions executable by the one or more processors, to cause the one or more processors to:
receive, from one or more sensors, sensor data associated with an environment of an autonomous vehicle;
generate, using an object detection model, object detection data based on the sensor data, the object detection data describing a location of an actor relative to the autonomous vehicle;
generate, using a graph neural network, motion forecast data associated with the actor, wherein the graph neural network comprises a plurality of nodes and a plurality of edges, the plurality of nodes indicative of a plurality of actors including the actor and the edges indicative of relationships among the plurality of actors,
wherein the graph neural network is configured to update at least one node state of a first node by passing at least one message between at least two nodes comprising the first node; and
determine a motion plan for the autonomous vehicle based on the motion forecast data; and
control the autonomous vehicle based on the motion plan.

### Claim 9 (depends on 8)

The vehicle computing system of claim 8, wherein the instructions cause the one or more processors to:
obtain map data associated with the environment of the autonomous vehicle;
concatenate, using the object detection model, the map data and the sensor data; and
generate, using the object detection model, the object detection data based on the map data and the sensor data.

### Claim 10 (depends on 8)

The vehicle computing system of claim 8, wherein t graph neural network is indicative of at least one of: (i) a partially connected graph neural network, or (ii) a fully connected graph neural network.

### Claim 11 (depends on 8)

The vehicle computing system of claim 8, wherein the at least one node state is indicative of at least one of: (i) a future trajectory of a respective actor of the plurality of actors or (ii) a behavior of the respective actor.

### Claim 12 (depends on 8)

The vehicle computing system of claim 8, wherein passing the at least one message between the at least two nodes is indicative of a potential interaction among the actors respectively associated with the at least two nodes.

### Claim 13 (depends on 12)

The vehicle computing system of claim 12, wherein the at least one message is indicative of a spatial relationship.

### Claim 14 (depends on 8)

The vehicle computing system of claim 8, wherein the sensor data comprises three-dimensional LiDAR data.

### Claim 15

A non-transitory computer-readable medium storing instructions executable by one or more processors of an autonomous vehicle computing system, to cause the one or more processors to:
receive, from one or more sensors, sensor data associated with an environment of an autonomous vehicle;
generate, using an object detection model, object detection data based on the sensor data, the object detection data describing a location of an actor relative to the autonomous vehicle;
generate, using a graph neural network, motion forecast data associated with the actor, wherein the graph neural network comprises a plurality of nodes and a plurality of edges, the plurality of nodes indicative of a plurality of actors including the actor and the edges indicative of relationships among the plurality of actors,
wherein the graph neural network is configured to update at least one node state of a first node by passing at least one message between at least two nodes comprising the first node; and
determine a motion plan for the autonomous vehicle based on the motion forecast data; and
control the autonomous vehicle based on the motion plan.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions cause the one or more processors to:
obtain map data associated with the environment of the autonomous vehicle;
concatenate, using the object detection model, the map data and the sensor data; and
generate, using the object detection model, the object detection data based on the map data and the sensor data.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the graph neural network is indicative of at least one of (i) a partially connected graph neural network, or (ii) a fully connected graph neural network.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the at least one node state is indicative of at least one of: (i) a future trajectory of a respective actor of the plurality of actors or (ii) a behavior of the respective actor.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein passing the at least one message between the at least two nodes is indicative of a potential interaction among the actors respectively associated with the at least two nodes.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the at least one message is indicative of a spatial relationship.