US 12,001,958 B2Granted

Future trajectory predictions in multi-actor environments for autonomous machine

AssigneeNVIDIA Corporation

InventorsAlexey Kamenev, Nikolai Smolyanskiy, Ishwar Kulkarni, Ollin Boer Bohan, Fangkai Yang, Alperen Degirmenci, Ruchi Bhargava, Urs Muller, David Nister, Rotem Aviv

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

In various examples, past location information corresponding to actors in an environment and map information may be applied to a deep neural network (DNN)—such as a recurrent neural network (RNN)—trained to compute information corresponding to future trajectories of the actors. The output of the DNN may include, for each future time slice the DNN is trained to predict, a confidence map representing a confidence for each pixel that an actor is present and a vector field representing locations of actors in confidence maps for prior time slices. The vector fields may thus be used to track an object through confidence maps for each future time slice to generate a predicted future trajectory for each actor. The predicted future trajectories, in addition to tracked past trajectories, may be used to generate full trajectories for the actors that may aid an ego-vehicle in navigating the environment.

Claims29

Dependent claims

A method comprising:

obtaining a first image representative of at least one past location of an actor within an environment;
obtaining a second image representative of map information associated with the environment;
computing, using one or more neural networks and based at least on image data representative of the first image and the second image, at least:
first data corresponding to a first map representing at least a first portion of the environment, one or more first pixels of the first map indicating at least a first future location in the environment at a first future time;
second data corresponding to a second map representing at least a second portion of the environment, one or more second pixels of the second map indicating at least a second future location in the environment at a second future time;
and
third data corresponding to a vector field associated with the second map;
determining that the second future location indicated by the second map is associated with the actor;
determining that one or more vectors from the vector field correspond to the second future location, the one or more vectors indicating a pixel distance between the second map and the first map;
determining, based at least on the one or more first pixels of the first map being associated with the pixel distance from the one or more second pixels of the second map, that the first future location indicated by the first map is associated with the second future location indicated by the second map;
and
determining, based at least on the first future location indicated by the first map being associated with the second future location indicated by the second map, that the first future location indicated by the first map is also associated with the actor.

9 more elements

A processor comprising:

one or more processors to:
compute, using one or more neural networks:
first data associated with a first map representing at least a first portion of an environment, the first map including one or more first pixels indicating a first future location, the first map being associated with a first future time;
second data associated with a second map representing at least a second portion of the environment, the second map including one or more second pixels indicating a second future location, the second map being associated with a second future time that is after the first future time;
and
third data representative of a vector field associated with the second map;
determine the second future location indicated by the second map is associated with an actor;
determine one or more vectors from the vector field corresponding to the second future location, the one or more vectors indicating a pixel distance between the second map and the first map;
determine based at least on the one or more first pixels from the first map being associated with the pixel distance from the one or more second pixels from the second map, that the second future location associated with the second future time is related to the first future location associated with the first future time that precedes the second future time;
and
determine based at least on the second future location being related to the first future location, that the first future location indicated by the first map is also associated with the actor.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/824,199Filed 19 Mar 2020
Published: US 2021/0295171 A123 Sep 2021
Granted: 4 Jun 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification16 codes

G06N 3/088
Learning methods: non-supervised learning, e.g. competitive learning
G06T 7/20
Image analysis: analysis of motion
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks
G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]

Family

US 2021/0295171 A1Pre-grant publication · published 23 Sep 2021
US 12,001,958 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key