Voice filtering other speakers from calls and audio messages

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a first instance of raw audio data corresponding to a voice-based command for an assistant-enabled device to facilitate an audio-based communication between a user of the assistant-enabled device and a recipient, the voice-based command spoken by the user and captured by the assistant-enabled device;
• receiving a second instance of the raw audio data corresponding to an utterance of audible contents for the audio-based communication spoken by the user and captured by the assistant-enabled device, the second instance of the raw audio data capturing one or more additional sounds that are not spoken by the user;
• executing a voice filtering recognition routine to determine whether to activate voice filtering for at least a voice of the user in the audio-based communication based on the first instance of the raw audio data,
• when the voice filtering recognition routine determines to activate voice filtering for at least the voice of the user:
• obtaining a respective speaker embedding of the user that represents voice characteristics for the user;
• and
• processing, using the respective speaker embedding of the user, the second instance of the raw audio data to generate enhanced audio data for the audio-based communication that isolates the utterance of the audible contents spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user;
• and
• transmitting the enhanced audio data to a recipient device associated with the recipient, the enhanced audio data when received by the recipient device, causing the recipient device to audibly output the utterance of the audible contents spoken by the user.