Mixed reality simulation for autonomous systems

A system comprising:

• a computer processor;
• and
• non-transitory computer readable medium for causing the computer processor to perform operations comprising:
• obtaining, from sensor data, map data of a geographic region and a plurality of trajectories of a plurality of agents located in the geographic region, wherein the plurality of agents and the map data have a corresponding physical location in the geographic region,
• generating, from the map data and the plurality of trajectories, a heterogeneous graph comprising a map layer and an agent layer, wherein generating the heterogeneous graph comprises:
• generating, in the map layer, plurality of map element nodes for a plurality of map elements defined in the map data, the plurality of map element nodes connected by a first plurality of edges based on relative positions between the plurality of map elements,
• processing the map layer by a map element encoder to add a plurality of map element encodings to the map layer of the heterogeneous graph,
• generating, in the agent layer, a plurality of agent nodes for the plurality of agents, the plurality of agent nodes connected by a second plurality of edges identifying relative positions of the plurality of agents, the plurality of agent nodes comprising an agent encoding, generated by an agent encoder, wherein the agent encoding encodes a plurality of relative historical positions of a corresponding agent with respect to a current position of the corresponding agent,
• connecting, to generate the heterogeneous graph, the plurality of agent nodes to the plurality of map element nodes based on the relative positions between the plurality of agents and the plurality of map elements,
• processing the agent layer and the map layer through an interaction encoder that adds a set of edges connecting the plurality of agent nodes to the plurality of map element nodes, and
• generating, by the interaction encoder executing a graph neural network, an interaction encoding by updating the plurality of map element encodings and the agent encoding of the plurality of agent nodes according to the set of edges,
• determining, for an agent of the plurality of agents, an agent route from a trajectory of the plurality of trajectories that corresponds to the agent,
• generating, by an agent attribute encoder, from the interaction encoding, an agent attribute encoding that encodes the agent and the agent route,
• processing the agent attribute encoding to generate positional information for the agent,
• updating the trajectory of the agent using the positional information to obtain an updated trajectory, and
• outputting the updated trajectory.

A non-transitory computer readable medium comprising computer readable program code for causing a computer system to perform operations comprising:

• obtaining, from sensor data, map data of a geographic region and a plurality of trajectories of a plurality of agents located in the geographic region, wherein the plurality of agents and the map data have a corresponding physical location in the geographic region;
• generating, from the map data and the plurality of trajectories, a heterogeneous graph comprising a map layer and an agent layer, wherein generating the heterogeneous graph comprises:
• generating, in the map layer, plurality of map element nodes for a plurality of map elements defined in the map data, the plurality of map element nodes connected by a first plurality of edges based on relative positions between the plurality of map elements,
• processing the map layer by a map element encoder to add a plurality of map element encodings to the map layer of the heterogeneous graph,
• generating, in the agent layer, a plurality of agent nodes for the plurality of agents, the plurality of agent nodes connected by a second plurality of edges identifying relative positions of the plurality of agents, the plurality of agent nodes comprising an agent encoding, generated by an agent encoder, wherein the agent encoding encodes a plurality of relative historical positions of a corresponding agent with respect to a current position of the corresponding agent,
• connecting, to generate the heterogeneous graph, the plurality of agent nodes to the plurality of map element nodes based on the relative positions between the plurality of agents and the plurality of map elements,
• processing the agent layer and the map layer through an interaction encoder that adds a set of edges connecting the plurality of agent nodes to the plurality of map element nodes, and
• generating, by the interaction encoder executing a graph neural network, an interaction encoding by updating the plurality of map element encodings and the agent encoding of the plurality of agent nodes according to the set of edges;
• determining, for an agent of the plurality of agents, an agent route from a trajectory of the plurality of trajectories that corresponds to the agent;
• generating, by an agent attribute encoder, from the interaction encoding, an agent attribute encoding that encodes the agent and the agent route;
• processing the agent attribute encoding to generate positional information for the agent;
• updating the trajectory of the agent using the positional information to obtain an updated trajectory;
• and
• outputting the updated trajectory.