Knowledge-driven scene priors for semantic audio-visual embodied navigation

A system for controlling navigation of a device in an environment using machine learning (ML) models, the system comprising:

• sensors configured to receive visual and audio observation data of the environment;
• a vision network configured to determine visual classification scores for objects and regions in the environment based on the visual observation data;
• a location predictor configured to determine (i) a distance and direction of a sounding object from the device based on the audio observation data and (ii) a direct-to-reverberant ratio (DRR) of an impulse sounding response between the sounding object and the device;
• an audio network configured to (i) determine audio classification scores for objects and regions in the environment based on the audio observation data and (ii) determine audio-semantic feature embeddings based at least in part on the visual classification scores, the determined distance and direction of the sounding object from the device, and the DRR, wherein the audio-semantic feature embeddings indicate spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment;
• a policy network configured to (i) encode visual information based on the visual classification scores and (ii) determine and output, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment;
• and
• one or more processing devices configured to control operation of the device based on the state representation.

A computing device configured to control navigation of a device in an environment using machine learning (ML) models, the computing device including a processing device configured to execute instructions stored in memory to:

• receive visual and audio observation data of the environment as sensed by the device;
• determine classification scores for objects and regions in the environment based on the visual and audio observation data;
• encode visual information based on the classification scores;
• determine (i) a distance and direction of a sounding object from the device based on the audio observation data and (ii) a direct-to-reverberant ratio (DRR) of an impulse sounding response between the sounding object and the device;
• determine audio-semantic feature embeddings based at least in part on the classification scores, the determined distance and direction of the sounding object from the device, and the DRR, wherein the audio-semantic feature embeddings indicate spatial relationships between objects in the environment, between regions in the environment, and between objects and regions in the environment;
• determine and output, based on the encoded visual information and the audio-semantic feature embeddings, a state representation corresponding to a state of the device within the environment;
• and
• control operation of the device based on the state representation.