System and method for completing trajectory prediction from agent-augmented environments

A system for completing trajectory prediction from agent-augmented environments, comprising:

• a memory storing instructions when executed by a processor cause the processor to:
• receive image data associated with surrounding environment of an ego agent, wherein the image data includes dynamic agents and static structures located within the surrounding environment of the ego agent;
• process an agent-augmented static representation which is an augmented representation of the surrounding environment of the ego agent based on the image data, including augmenting boundaries of the static structures with respective augmented static agents that represent an obstruction to respective prospective paths of the dynamic agents that are caused by the existence of the respective static structures, and represent at least one of the static structures as multiple augmented static agents to enable learning of physical constraints of the surrounding environment of the ego agent and predict implausible respective future paths of the dynamic agents in view of the physical constraints of the surrounding environment;
• process the agent-augmented static representation of the surrounding environment of the ego agent includes augmenting boundaries of the static agents with augmented static agents that represent an occlusion to respective paths of the dynamic agents that are caused by the static structures, and leveraging a similarity between at least one static agent and at least one static structures to represent environmental structures as multiple augmented static agents to further enable learning of physical constraints of the surrounding environment of the ego agent;
• process a set of spatial graphs that correspond to an observation time horizon based on the agent-augmented static representation, wherein nodes in the spatial graphs are simultaneously clustered considering their spatial and temporal interactions;
• predict future trajectories of agents that are located within the surrounding environment of the ego agent based on the spatial graphs;
• and
• controlling at least one component of the ego agent based on the predicted future trajectories of agents within the surrounding environment of the ego agent to avoid overlap with the respective predicted future trajectories of the agents.

A non-transitory computer readable storage medium storing instructions that when executed by a computer, which includes a processor perform a method, the method comprising:

• receiving image data associated with surrounding environment of an ego agent, wherein the image data includes dynamic agents and static structures located within the surrounding environment of the ego agent;
• processing an agent-augmented static representation which is an augmented representation of the surrounding environment of the ego agent based on the image data, including augmenting boundaries of the static structures with respective augmented static agents that represent an obstruction to respective prospective paths of the dynamic agents that are caused by the existence of the static structures, further including representing at least one of the static structures as multiple augmented static agents enabling learning of physical constraints of the surrounding environment of the ego agent and predicting implausible respective future paths of the dynamic agents in view of the physical constraints of the surrounding environment;
• processing the agent-augmented static representation of the surrounding environment of the ego agent includes augmenting boundaries of the static agents with augmented static agents that represent an occlusion to respective paths of the dynamic agents that are caused by the static structures, and leveraging a similarity between at least one static agent and at least one static structures to represent environmental structures as multiple augmented static agents to further enable learning of physical constraints of the surrounding environment of the ego agent;
• processing a set of spatial graphs that correspond to an observation time horizon based on the agent-augmented static representation, wherein nodes in the spatial graphs are simultaneously clustered considering their spatial and temporal interactions;
• predicting future trajectories of agents that are located within the surrounding environment of the ego agent based on the spatial graphs;
• and
• controlling at least one component of the ego agent based on the predicted future trajectories of agents within the surrounding environment of the ego agent to avoid overlap with the respective predicted future trajectories of the agents.