# US 12,112,622 B2 · Systems and methods for heterogeneous multi-agent multi-modal trajectory prediction with evolving interaction graphs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,622 B2 - Document ID: US12112622B2 - Status: Granted - Kind: B2 - Application number: 17/024,080 - Filed: 2020-09-17 - Published: 2021-09-16 - Granted: 2024-10-08 - Earliest claimed: 2020-03-12 - Assignees: Honda Motor Co., Ltd. - Applicants: Honda Motor Co., Ltd. - Inventors: Jiachen Li; Chiho Choi - Web page: https://patentgrep.com/patent/US12112622B2 ## CPC - G08G 1/0133: {for classifying traffic situation} - G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/09: Supervised learning - G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]} - G08G 1/0129: {for creating historical data or processing based on historical data} - G08G 1/0141: {for traffic information dissemination} - G08G 1/015: with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles - G08G 1/164: {Centralised systems, e.g. external to vehicles} - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The systems and methods herein utilize interactive Gaussian processes for crowd navigation. For example, an encoder receives sensor data and context information. The encoder also extracts interaction patterns from observed trajectories from the sensor data and context information. The encoder further generates a static latent interaction graph for a first time step based on the interaction patterns. A recurrent module generates a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph. The series of time steps are separated by a re-encoding gap. The decoder generates multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs. ## Claims ### Claim 1 A system for heterogeneous multi-agent multi-modal trajectory prediction, comprising: a processor; an encoder, implemented via the processor, configured to iteratively perform an encoding process for a plurality of time steps defined as a re-encoding gap, the encoding process causing the processor to: receive sensor data and context information as context data in a form of an observation graph having agent nodes bidirectionally connected to each other and at least one context node having outgoing edges to each of the agent nodes; extract interaction patterns from observed trajectories from the context data; and generate a static latent interaction graph for a first time step based on the interaction patterns; a recurrent module, implemented via the processor, configured to generate a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph, wherein time steps of the series of time steps are separated by the re-encoding gap; and a decoder, implemented via the processor, configured to generate a multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs. ### Claim 2 (depends on 1) The system of claim 1, wherein the distribution of time dependent static latent interaction graphs is generated based on an observation graph based on sensor data at a corresponding time step of the series of time steps. ### Claim 3 (depends on 1) The system of claim 1, wherein the distribution of time dependent static latent interaction graphs is a Gaussian distribution over the series of time steps. ### Claim 4 (depends on 1) The system of claim 1, wherein the decoder operates in a first stage and a second stage for the distribution of time dependent static latent interaction graphs, wherein the first stage is based on a historical time horizon and the second stage is based on a forecasting time horizon. ### Claim 5 (depends on 4) The system of claim 4, wherein in the first stage the decoder is configured to use ground-truth states at the first time step corresponding to the static latent interaction graph. ### Claim 6 (depends on 4) The system of claim 4, wherein in the second stage the decoder is configured to use prediction hypothesis at a next time step of the series of time steps. ### Claim 7 (depends on 1) The system of claim 1, wherein the decoder is further configured to output weights for the multi-modal distribution of future states based on probabilities of a next action for each modality represented in the sensor data. ### Claim 8 A method for heterogeneous multi-agent multi-modal trajectory prediction, comprising: performing an encoding process, iteratively, for a plurality of time steps defined as a re-encoding gap, wherein the encoding process includes receiving sensor data and context information as context data, the context data being in a form of an observation graph having agent nodes bidirectionally connected to each other and at least one context node having outgoing edges to each of the agent nodes, associated with a number of heterogeneous agents, extracting interaction patterns from observed trajectories from context data, and generating a static latent interaction graph for a first time step based on the interaction patterns; generating a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph, wherein time steps of the series of time steps are separated by the re-encoding gap; and generating a multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs. ### Claim 9 (depends on 8) The method of claim 8, wherein the distribution of time dependent static latent interaction graphs is generated based on an observation graph based on sensor data at a corresponding time step of the series of time steps. ### Claim 10 (depends on 8) The method of claim 8, wherein the distribution of time dependent static latent interaction graphs is a Gaussian distribution over the series of time steps. ### Claim 11 (depends on 8) The method of claim 8, wherein generating the multi-modal distribution of future states includes a first stage and a second stage, and wherein the first stage is based on a historical time horizon and the second stage is based on a forecasting time horizon. ### Claim 12 (depends on 11) The method of claim 11, wherein the historical time horizon is based on ground-truth states at the first time step corresponding to the static latent interaction graph, and wherein the forecasting time horizon is based on a prediction hypothesis at a next time step of the series of time steps. ### Claim 13 (depends on 8) The method of claim 8, the method further comprising: outputting weights for the multi-modal distribution of future states based on probabilities of a next action for each modality represented in the sensor data. ### Claim 14 A non-transitory computer readable storage medium storing instructions that when executed by a computer having a processor to perform a method for heterogeneous multi-agent multi-modal trajectory prediction, the method comprising: performing an encoding process, iteratively, for a plurality of time steps defined as a re-encoding gap, wherein the encoding process includes receiving sensor data and context information as context data, the context data being in a form of an observation graph having agent nodes bidirectionally connected to each other and at least one context node having outgoing edges to each of the agent nodes, associated with a number of heterogeneous agents, extracting interaction patterns from observed trajectories from the context data, and generating a static latent interaction graph for a first time step based on the interaction patterns; generating a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph, wherein time steps of the series of time steps are separated by the re-encoding gap; and generating a multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs. ### Claim 15 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein the distribution of time dependent static latent interaction graphs is generated based on an observation graph based on sensor data at a corresponding time step of the series of time steps. ### Claim 16 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein the distribution of time dependent static latent interaction graphs is a Gaussian distribution over the series of time steps. ### Claim 17 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein generating the multi-modal distribution of future states includes a first stage and a second stage, and wherein the first stage is based on a historical time horizon and the second stage is based on a forecasting time horizon. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the historical time horizon is based on ground-truth states at the first time step corresponding to the static latent interaction graph, and wherein the forecasting time horizon is based on a prediction hypothesis at a next time step of the series of time steps. ### Claim 19 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein the method further comprises outputting weights for the multi-modal distribution of future states based on probabilities of a next action for each modality represented in the sensor data.