Method and system for scene-aware interaction

A navigation system for providing driving instructions to a driver of a vehicle traveling on a route based on real-time description of objects in a scene pertinent to the route of the vehicle, wherein the navigation system is operatively connected through one or a combination of wired and wireless communication channels to multiple sensors configured to provide measurements of the scene and an output device configured to communicate the driving instructions to the driver of the vehicle, the navigation system comprising:

• an input interface configured to accept first measurements sensed by a first sensor of the multiple sensors and second measurements sensed by a second sensor of the multiple sensors;
• a memory configured to store executable instructions;
• a processor configured to execute the executable instructions to generate the driving instructions for the vehicle by performing a computer-implemented multimodal fusion method, wherein to perform the computer-implemented multimodal fusion method the processor is configured to:
• extract first features indicative of first attributes and first spatial relationships of first objects sensed by the first sensor, based on the first measurements;
• extract second features indicative of second attributes and second spatial relationships of second objects sensed by the second sensor, based on the second measurements;
• annotate the first features with encodings of a first direction for the vehicle to follow the route with respect to a first corresponding object of the first objects and the second features with encodings of a second direction for the vehicle to follow the route with respect to a second corresponding object of the second objects;
• encode the annotated first features and the annotated second features with a multimodal attention neural network by temporally correlating the annotated first features of the first measurements sensed at different instances of time to produce first encodings, temporally correlating the annotated second features of the second measurements sensed at different instances of time to produce second encodings, transforming the first encodings and the second encodings into a common latent space, and fusing the transformed first encodings and the transformed second encodings using an attention mechanism producing an encoded representation of the scene including a weighted combination of the first encodings and the second encodings with weights determined by the attention mechanism;
• decode the encoded representation of the scene with a sentence generation neural network to generate a driving instruction of the driving instructions using a vocabulary of types of salient objects, properties of the salient objects, and navigating actions;
• and
• submit the driving instruction of the driving instructions to the output device.