Joint Acoustic Echo Cancellation (AEC) and Personalized Noise Suppression (PNS)

A method implemented in a data processing system for processing audio signals, the method comprising:

• obtaining first embeddings representing an encoded version of a far-end signal associated with a first computing device participating in an online communication session;
• obtaining second embeddings representing an encoded version of a near-end signal associated with a second computing device participating in the online communication session, the near-end signal including speech of a target speaker, one or more interfering speakers, and an echo signal;
• obtaining target speaker embeddings representing speech characteristics of the target speaker;
• providing the first embeddings, the second embeddings, and the target speaker embeddings as an input to a machine learning model comprising:
• a self-attention alignment block configured to use attention to determine a soft-alignment of first features of the first embeddings and second features of the second embeddings, the self-attention alignment block outputting attention weights associated with the soft-alignment;
• a first fusing block configured to fuse the first embeddings, the second embeddings, and the attention weights into a first fused output;
• a first group of Long Short-Term Memory (LSTM) blocks trained to perform acoustic echo cancellation (AEC) on the first fused output to remove the echo signal and to output third features,
• a second fusing block configured to fuse the third features output by the first group of LSTM blocks and the target speaker embeddings to generate a second fused output;
• and
• a second group of LSTM blocks trained to perform personalized noise suppression (PNS) on the second fused output to remove the speech from the one or more interfering speakers;
• and
• obtaining an audio signal comprising the speech of the target speaker from the machine learning model.

A data processing system comprising:

• a processor;
• and
• a machine-readable medium storing executable instructions that, when executed, cause the processor to perform operations comprising:
• instantiating an instance of a machine learning model comprising:
• a self-attention alignment block configured to use attention to determine a soft-alignment of first features of first embeddings and second features of second embeddings, the first embeddings representing an encoded version of a far-end signal associated with a first computing device participating in an online communication session, second embeddings representing an encoded version of a near-end signal associated with a second computing device participating in the online communication session, the near-end signal including speech of a target speaker, one or more interfering speakers, and an echo signal, the self-attention alignment block outputting attention weights associated with the soft-alignment;
• a first fusing configured to the first embeddings, the second embeddings, and the attention weights into a first fused output;
• a first group of Long Short-Term Memory (LSTM) blocks configured to perform acoustic echo cancellation (AEC) on the first fused output to remove the echo signal and to output third features,
• a second fusing block configured to fuse the third features output by the first group of LSTM blocks and target speaker embeddings representing speech characteristics of the target speaker into a second fused output, and a second group of LSTM blocks configured to perform personalized noise suppression (PNS) on the second fused output to remove the speech from the one or more interfering speakers;
• and
• training the machine learning model to perform AEC and PNS using successive batches of training data by:
• training the machine learning model with a first batch of training data for training the machine learning model to perform AEC, the first batch of training data comprising target speaker data, noise data, echo data, and a speaker embeddings vector, the speaker embeddings vector is empty to cause the machine learning model to bypass the second group of LSTM blocks after the first group of LSTM blocks processes the first batch of training data;
• training the machine learning model with a second batch of training data for training the machine learning model to perform PNS, the second batch of training data including target speaker data, interfering speaker data, and noise data, the speaker embeddings vector in the second batch of training data including features representing speech characteristics the target speaker;
• and
• training the machine learning model with a third batch of training data for training the machine learning model to jointly perform both AEC and PNS, the third batch of training data comprising target speaker data, interfering speaker data, noise data, and echo data.