Systems and methods for multi-modal visual reasoning using dynamic scene graphs and knowledge graphs

A method comprising:

• obtaining first sensor data associated with a vehicle operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor;
• determining a set of features associated with the environment based on the first sensor data, the set of features comprising one or more objects and one or more agents;
• in response to determining the set of features, generating a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment;
• in response to generating the scene graph, generating a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the vehicle as represented by the first plurality of nodes of the scene graph;
• generating a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment;
• revising the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph;
• generating a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph;
• and
• providing the control signal to the vehicle to cause the adjusted operation of the vehicle.

One or more non-transitory computer-readable mediums storing instructions thereon that, when executed by one or more processors, cause the one or more processors to:

• obtain first sensor data associated with a device operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor;
• determine a set of features associated with the environment based on the sensor data, the set of features comprising one or more objects and one or more agents;
• in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment;
• in response to generating the scene graph, generate a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the device as represented by the first plurality of nodes of the scene graph;
• generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment;
• revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph;
• generate a control signal configured to adjust an operation of the device based on updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph;
• and
• provide the control signal to the device to cause the adjusted operation of the device.