Audio-visual speech separation

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• obtaining, for each frame in a stream of frames from a video in which faces of one or more speakers have been detected, a respective per-frame face embedding of a face of each speaker;
• processing, for each speaker, the per-frame face embeddings of the face of the speaker using a video processing neural network to generate visual features for the face of the speaker;
• obtaining a spectrogram of an audio soundtrack for the video;
• processing the spectrogram using an audio processing neural network to generate an audio embedding for the audio soundtrack;
• combining the visual features for the faces of the one or more speakers and the audio embedding for the audio soundtrack to generate an audio-visual embedding for the video, wherein the audio-visual embedding represents both audio features of the audio soundtrack and visual features of the respective faces of the one or more speakers;
• processing, using a masking neural network, the audio-visual embedding for the video that represents both the audio features of the audio soundtrack and the visual features for the faces of the one or more speakers to generate a respective spectrogram mask for each of the one or more speakers, wherein each spectrogram mask describes time-frequency relationships between clean speech for a respective speaker and background interference in a spectrogram of the audio soundtrack;
• and
• determining, using the respective spectrogram masks and the corresponding audio soundtrack, a respective isolated speech spectrogram for each speaker that isolates the speech of the speaker in the video.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers, cause the one or more computers to perform operations comprising:

• obtaining, for each frame in a stream of frames from a video in which faces of one or more speakers have been detected, a respective per-frame face embedding of a face of each speaker;
• processing, for each speaker, the per-frame face embeddings of the face of the speaker using a video processing neural network to generate visual features for the face of the speaker;
• obtaining a spectrogram of an audio soundtrack for the video;
• processing the spectrogram using an audio processing neural network to generate an audio embedding for the audio soundtrack;
• combining the visual features for the faces of the one or more speakers and the audio embedding for the audio soundtrack to generate an audio-visual embedding for the video, wherein the audio-visual embedding represents both audio features of the audio soundtrack and visual features of the respective faces of the one or more speakers;
• processing, using a masking neural network, the audio-visual embedding for the video that represents both the audio features of the audio soundtrack and the visual features for the faces of the one or more speakers to generate a respective spectrogram mask for each of the one or more speakers, wherein each spectrogram mask describes time-frequency relationships between clean speech for a respective speaker and background interference in a spectrogram of the audio soundtrack;
• and
• determining, using the respective spectrogram masks and the corresponding audio soundtrack, a respective isolated speech spectrogram for each speaker that isolates the speech of the speaker in the video.