Mobile robot with audio perception system

A mobile robot comprising:

• a microphone array;
• one or more processors in data communication with the microphone array;
• and
• one or more memory in data communication with the one or more processors, the one or more memory including computer readable data stored thereon that, when executed by the one or more processors, causes the processor to perform a method that includes
• receiving audio signals via the microphone array, the microphone array including a set of microphones at least partially disposed on the mobile robot;
• extracting audio feature data of acoustic activity from the audio signals;
• generating Direction-of-Arrival (DOA) data of the acoustic activity based on the audio signals;
• generating, via at least one pretrained machine learning model, audio event data using the audio feature data, the audio event data identifying at least one sound source of the audio feature data;
• extracting entity data by using the audio event data to query a knowledge graph, the entity data having a relation with the audio event data, the entity data including location data that identifies a location of the at least one sound source within the environment;
• generating semantic audio scene data using the audio event data, the DOA data, and the entity data, the semantic audio scene data including an association between the at least one sound source, a direction of the at least one sound source relative to the mobile robot, and the location data that identifies the location of the at least one sound source;
• and
• performing an action of the mobile robot based on the semantic audio scene data.

One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform a method for controlling a mobile robot in an environment, the method comprising:

• receiving audio signals via a microphone array, the microphone array including a set of microphones at least partially disposed on the mobile robot;
• extracting audio feature data of acoustic activity from the audio signals;
• generating Direction-of-Arrival (DOA) data of the acoustic activity based on the audio signals;
• generating, via at least one pretrained machine learning model, audio event data using the audio feature data, the audio event data identifying at least one sound source of the audio feature data;
• extracting entity data by using the audio event data to query a knowledge graph, the entity data having a relation with the audio event data, the entity data including location data that identifies a location of the at least one sound source within the environment;
• generating semantic audio scene data using the audio event data, the DOA data, and the entity data, the semantic audio scene data including an association between the at least one sound source, a direction of the at least one sound source relative to the mobile robot, and the location data that identifies the location of the at least one sound source;
• and
• performing an action of the mobile robot based on the semantic audio scene data.