Systems and methods for multi-modal visual reasoning using knowledge graphs maintained over a period of time

A method for processing multi-modal data representing an environment during automated operation of a vehicle, the method comprising:

• obtaining first sensor data associated with a vehicle operating in an environment at a first point in time;
• generating a knowledge graph based on the first sensor data, the knowledge graph comprising (1) a plurality of nodes representing a plurality of objects and one or more agents in the environment at the first point in time, and (2) edges between respective pairs of the plurality of nodes representing relationships agents in the environment at the first point in time;
• in response to obtaining second sensor data associated with the vehicle at a second point in time, determining one or more changes to the relationships involving the objects and the agents;
• updating the knowledge graph based on the one or more changes to the relationships by incrementally modifying the edges and the plurality of nodes based on the determined one or more changes to the relationships involving the objects and the agents;
• and
• in response to updating the knowledge graph, providing a control signal to the vehicle to cause operation of a vehicle based on determining the modification to the edges between the respective pairs of the plurality of nodes of the updated knowledge graph cause the updated knowledge graph to define a predefined relationship pattern.

One or more non-transitory computer-readable mediums storing instructions thereon that, when executed by one or more processors, cause the one or more processors to:

• obtain first sensor data associated with a device operating in an environment at a first point in time;
• generate a knowledge graph based on the first sensor data, the knowledge graph comprising (1) a plurality of nodes representing a plurality of objects and one or more agents in the environment at the first point in time, and (2) edges between respective pairs of the plurality of nodes representing relationships between objects or agents of the respective pairs of the plurality of nodes in the environment at the first point in time;
• in response to obtaining second sensor data associated with the device at a second point in time, determine one or more changes to the relationships involving the objects and the agents;
• update the knowledge graph based on the one or more changes to the relationships by incrementally modifying the edges and the plurality of nodes based on the determined one or more changes to the relationships involving the objects and the agents;
• and
• in response to updating the knowledge graph, provide a control signal to the device to cause operation of a device based on determining the modification to the edges between the respective pairs of the plurality of nodes of the updated knowledge graph cause the updated knowledge graph to define a predefined relationship pattern.