# US 2021/0276595 A1 · Systems and Methods for Latent Distribution Modeling for Scene-Consistent Motion Forecasting > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0276595 A1 - Document ID: US20210276595A1 - Status: Published application - Kind: A1 - Application number: 17/150,995 - Filed: 2021-01-15 - Published: 2021-09-09 - Earliest claimed: 2020-03-05 - Assignees: Uber Technologies, Inc. - Applicants: Uber Technologies, Inc. - Inventors: Sergio Casas; Cole Chistian Gulino; Shun Da Suo; Katie Z. Luo; Renjie Liao; Raquel Urtasun - Web page: https://patentgrep.com/patent/US20210276595A1 ## CPC - G06V 10/82: using neural networks - B60W 60/0027: {using trajectory prediction for other traffic participants} - G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)} - G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 30/19173: {Classification techniques} - G08G 1/165: {for passive traffic, e.g. including static obstacles, trees} - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - G08G 1/202: {Dispatching vehicles on the basis of a location, e.g. taxi dispatching} - G06V 30/2504: {Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract A computer-implemented method for determining scene-consistent motion forecasts from sensor data can include obtaining scene data including one or more actor features. The computer-implemented method can include providing the scene data to a latent prior model, the latent prior model configured to generate scene latent data in response to receipt of scene data, the scene latent data including one or more latent variables. The computer-implemented method can include obtaining the scene latent data from the latent prior model. The computer-implemented method can include sampling latent sample data from the scene latent data. The computer-implemented method can include providing the latent sample data to a decoder model, the decoder model configured to decode the latent sample data into a motion forecast including one or more predicted trajectories of the one or more actor features. The computer-implemented method can include receiving the motion forecast including one or more predicted trajectories of the one or more actor features from the decoder model. ## Claims ### Claim 1 A computer-implemented method for determining scene-consistent motion forecasts from sensor data, the method comprising: obtaining, by a computing system comprising one or more computing devices, scene data comprising one or more actor features; providing, by the computing system, the scene data to a latent prior model, the latent prior model configured to generate scene latent data in response to receipt of scene data, the scene latent data comprising one or more latent variables; obtaining, by the computing system, the scene latent data from the latent prior model; sampling, by the computing system, latent sample data from the scene latent data; providing, by the computing system, the latent sample data to a decoder model, the decoder model configured to decode the latent sample data into a motion forecast comprising one or more predicted trajectories of the one or more actor features; and receiving, by the computing system, the motion forecast comprising one or more predicted trajectories of the one or more actor features from the decoder model. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the decoder model comprises a deterministic decoder model. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the decoder model comprises a specified and tractable conditional likelihood. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the one or more latent variables are respective to the one or more actor features such that each actor feature has an associated latent variable of the scene latent data that is anchored to the actor feature. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the one or more latent variables comprise one or more continuous latent variables. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: obtaining, by the computing system, one or more scene observations; providing, by the computing system, the one or more scene observations to a scene feature extraction model, the scene feature extraction model comprising one or more neural networks configured to extract one or more scene features from the one or more scene observations; receiving, by the computing system, the one or more scene features from the scene feature extraction model; providing, by the computing system, the one or more scene features to an actor feature recognition model, the actor feature recognition model configured to: extract spatial feature maps for bounding boxes from the one or more scene features by rotated region of interest align; pool a region around each spatial feature map to produce pooled actor features; downsample the pooled actor features by applying one or more downsampling convolutional neural networks; and max-pool along spatial dimensions to reduce each pooled actor feature to a respective actor feature of the one or more actor features; and receiving, by the computing system, the one or more actor features from the actor feature recognition model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: sampling, by the computing system, second latent sample data from the scene latent data; providing, by the computing system, the second latent sample data to the decoder model; and receiving, by the computing system, a second motion forecast comprising one or more second predicted trajectories of the one or more actor features from the decoder model. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein at least one of the latent prior model or the decoder model comprises a scene interaction model configured to model the latent distribution as an interaction graph comprising one or more nodes representative of the one or more actor features and one or more edges representative of interactions between the one or more actor features. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the scene interaction model comprises one or more graph neural networks. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein a message function of the one or more graph neural networks comprises a multi-layer perceptron model that takes as input one or more terminal nodes of the one or more nodes at a previous propagation step of the one or more graph neural networks. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, wherein an aggregation function of the one or more graph neural networks comprises a feature-wise max-pooling aggregation function. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, wherein a gated recurrent unit cell is configured to update a state of the one or more nodes. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the one or more actor features comprise data descriptive of a context of one or more traffic participants. ### Claim 14 A computer-implemented method of training a motion forecasting system, the method comprising: obtaining, by a computing system comprising one or more computing devices, a training dataset comprising one or more training examples labeled with ground truth data, the one or more training examples comprising one or more actor features and the ground truth data comprising a ground truth context of the one or more actor features; providing, by the computing system, the one or more training examples labeled with ground truth data to a latent encoder model, the latent encoder model configured to produce a first latent distribution in response to receipt of the one or more training examples and the ground truth data; providing, by the computing system, the one or more training examples to a latent prior model, the latent prior model configured to produce a second latent distribution in response to receipt of the one or more training examples; determining, by the computing system, a training loss based at least in part on the first latent distribution and the second latent distribution; and backpropagating, by the computing system, the training loss through at least the latent prior model to train at least the latent prior model. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein the training loss comprises a KL divergence loss between the first latent distribution and the second latent distribution. ### Claim 16 (depends on 14) The computer-implemented method of claim 14, wherein the method further comprises: providing, by the computing system, training scene observations to a feature extraction model; receiving, by the computing system, one or more predicted features from the feature extraction model; and determining, by the computing system, a feature loss between the one or more predicted features and the ground truth data; wherein the training loss comprises the feature loss. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, wherein the feature loss comprises a cross-entropy loss between the one or more predicted features and one or more training features of the ground truth data and a regression loss between bounding boxes of the one or more predicted features and bounding boxes of the ground truth data. ### Claim 18 (depends on 14) The computer-implemented method of claim 14, further comprising: sampling, by the computing system, the first latent distribution to produce one or more first latent samples; sampling, by the computing system, the second latent distribution to produce one or more second latent samples; providing, by the computing system, the one or more first latent samples to a decoder model; receiving, by the computing system, one or more first predicted trajectories from the decoder model; providing, by the computing system, the one or more second latent samples to the decoder model; receiving, by the computing system, one or more second predicted trajectories from the decoder model; and determining, by the computing system, a reconstruction loss based at least in part on the one or more first predicted trajectories and the one or more second predicted trajectories; wherein the training loss comprises the reconstruction loss. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein the reconstruction loss comprises a Huber loss. ### Claim 20 A computer-implemented method of operating an autonomous vehicle, the method comprising: obtaining, by the computing system, one or more scene observations; providing, by the computing system, the one or more scene observations to a feature extraction model, the feature extraction model configured to produce scene data comprising one or more actor features from the one or more scene observations; receiving, by the computing system, the scene data comprising one or more actor features from the feature extraction model; providing, by the computing system, the scene data to a latent prior model, the latent prior model configured to generate scene latent data in response to receipt of scene data, the scene latent data comprising one or more latent variables; obtaining, by the computing system, the scene latent data from the latent prior model; sampling, by the computing system, one or more latent samples from the scene latent data; providing, by the computing system, the one or more latent samples to a decoder model, the decoder model configured to decode the latent samples into a motion forecast comprising one or more predicted trajectories of the one or more actor features; obtaining, by the computing system, one or more motion forecasts comprising one or more predicted trajectories of the one or more actor features from the decoder model; providing, by the computing system, the one or more predicted trajectories to a motion planning model configured to generate a motion plan for an autonomous vehicle based at least in part on the one or more predicted trajectories; and implementing, by the computing system, the motion plan to control the autonomous vehicle.