Systems and methods for heterogeneous multi-agent multi-modal trajectory prediction with evolving interaction graphs

A method for heterogeneous multi-agent multi-modal trajectory prediction, comprising:

• performing an encoding process, iteratively, for a plurality of time steps defined as a re-encoding gap, wherein the encoding process includes receiving sensor data and context information as context data, the context data being in a form of an observation graph having agent nodes bidirectionally connected to each other and at least one context node having outgoing edges to each of the agent nodes, associated with a number of heterogeneous agents, extracting interaction patterns from observed trajectories from context data, and generating a static latent interaction graph for a first time step based on the interaction patterns;
• generating a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph, wherein time steps of the series of time steps are separated by the re-encoding gap;
• and
• generating a multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs.

A non-transitory computer readable storage medium storing instructions that when executed by a computer having a processor to perform a method for heterogeneous multi-agent multi-modal trajectory prediction, the method comprising:

• performing an encoding process, iteratively, for a plurality of time steps defined as a re-encoding gap, wherein the encoding process includes receiving sensor data and context information as context data, the context data being in a form of an observation graph having agent nodes bidirectionally connected to each other and at least one context node having outgoing edges to each of the agent nodes, associated with a number of heterogeneous agents, extracting interaction patterns from observed trajectories from the context data, and generating a static latent interaction graph for a first time step based on the interaction patterns;
• generating a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph, wherein time steps of the series of time steps are separated by the re-encoding gap;
• and
• generating a multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs.