Audio-visual separation of on-screen sounds based on machine learning models

A computing device, comprising:

• one or more processors;
• and
• data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out operations comprising:
• receiving, by the computing device, an audio waveform associated with a plurality of video frames of video content;
• determining, by a time-domain convolutional masking network of a neural network and from the audio waveform, a neural representation comprising one or more audio frames, wherein each audio frame of the one or more audio frames comprises a respective plurality of coefficients, and wherein the respective plurality of coefficients represent one or more audio features in an encoded mixture of the audio waveform;
• predicting, by the neural network and based on the neural representation, one or more audio sources associated with the plurality of video frames;
• modifying the one or more predicted audio sources by applying a mixture consistency projection that constrains the one or more predicted audio sources to add up to the received audio waveform;
• and
• providing, by the computing device, the modified one or more predicted audio sources.

An article of manufacture comprising one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out operations comprising:

• receiving, by the computing device, an audio waveform associated with a plurality of video frames of video content;
• determining, by a time-domain convolutional masking network of a neural network and from the audio waveform, a neural representation comprising one or more audio frames, wherein each audio frame of the one or more audio frames comprises a respective plurality of coefficients, and wherein the respective plurality of coefficients represent one or more audio features in an encoded mixture of the audio waveform;
• predicting, by the neural network and based on the neural representation, one or more audio sources associated with the plurality of video frames;
• modifying the one or more predicted audio sources by applying a mixture consistency projection that constrains the one or more predicted audio sources to add up to the received audio waveform;
• and
• providing, by the computing device, the modified one or more predicted audio sources.