# US 12,217,768 B2 · Audio-visual separation of on-screen sounds based on machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,768 B2 - Document ID: US12217768B2 - Status: Granted - Kind: B2 - Application number: 18/226,545 - Filed: 2023-07-26 - Published: 2023-11-30 - Granted: 2025-02-04 - Earliest claimed: 2021-03-26 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Efthymios Tzinis; Scott Wisdom; Aren Jansen; John R. Hershey - Web page: https://patentgrep.com/patent/US12217768B2 ## CPC - G10L 25/57: for processing of video signals - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G10L 21/0308: characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques - G10L 25/30: using neural networks - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Apparatus and methods related to separation of audio sources are provided. The method includes receiving an audio waveform associated with a plurality of video frames. The method includes estimating, by a neural network, one or more audio sources associated with the plurality of video frames. The method includes generating, by the neural network, one or more audio embeddings corresponding to the one or more estimated audio sources. The method includes determining, based on the audio embeddings and a video embedding, whether one or more audio sources of the one or more estimated audio sources correspond to objects in the plurality of video frames. The method includes predicting, by the neural network and based on the one or more audio embeddings and the video embedding, a version of the audio waveform comprising audio sources that correspond to objects in the plurality of video frames. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving, by a computing device, an audio waveform associated with a plurality of video frames of video content; determining, by a time-domain convolutional masking network of a neural network and from the audio waveform, a neural representation comprising one or more audio frames, wherein each audio frame of the one or more audio frames comprises a respective plurality of coefficients, and wherein the respective plurality of coefficients represent one or more audio features in an encoded mixture of the audio waveform; predicting, by the neural network and based on the neural representation, one or more audio sources associated with the plurality of video frames; modifying the one or more predicted audio sources by applying a mixture consistency projection that constrains the one or more predicted audio sources to add up to the received audio waveform; and providing, by the computing device, the modified one or more predicted audio sources. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: predicting, for a given audio frame of the one or more audio frames, a respective mask, and wherein the predicting of the one or more audio sources comprises applying an inverse transform to a result of multiplying the respective mask with the plurality of coefficients corresponding to the given audio frame. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: extracting, by a video embedding network, one or more visual features from each of the plurality of video frames; and generating, for each of the plurality of video frames, a video embedding based on the one or more visual features. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising: generating, by the video embedding network, a global video embedding comprising a global representation of the one or more visual features, and a plurality of spatio-temporal locations of the video content. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, further comprising: generating, by the video embedding network, a local video embedding comprising, for each video frame of the plurality of video frames, a temporal representation of the one or more visual features in the video frame. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: generating, by the video embedding network, a global video embedding based on a plurality of local video embeddings corresponding to the plurality of video frames. ### Claim 7 (depends on 3) The computer-implemented method of claim 3, wherein the video embedding network comprises a plurality of stacked two-dimensional (2D) separable dilated convolutional blocks and a dense layer at an end of the plurality of blocks. ### Claim 8 (depends on 3) The computer-implemented method of claim 3, further comprising: generating, by the neural network, one or more audio embeddings corresponding to the one or more predicted audio sources; generating, by the neural network and for each audio embedding corresponding to the one or more predicted audio sources and based on the video embedding, a spatio-temporal audio-visual embedding based on an attention operation that aligns the one or more predicted audio sources with spatio-temporal positions of on-screen objects in the plurality of video frames; determining, by the neural network and based on the spatio-temporal audio-visual embedding, whether a given audio source of the one or more estimated audio sources corresponds to an on-screen object in the plurality of video frames; and predicting, by the neural network, a version of the audio waveform comprising audio sources that correspond to the on-screen objects in the plurality of video frames. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the neural network comprises a classifier, wherein a first attention operation is applied to generate the one or more audio embeddings, a second attention operation is applied to generate the video embedding, and wherein the determining of whether the given audio source corresponds to the on-screen object in the plurality of video frames comprises applying the classifier based on the one or more audio embeddings and the video embedding. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein the neural network comprises a classifier, and the attention operation is applied to the one or more audio embeddings and the video embedding, to produce a representation, wherein the determining of whether the given audio source corresponds to the on-screen object in the plurality of video frames comprises applying the classifier based on the representation. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing device, a request to provide the one or more predicted audio sources; sending the request from the computing device to a second computing device, the second computing device comprising a trained version of the neural network; after sending the request, the computing device receiving, from the second computing device, the one or more predicted audio sources; and outputting, by the computing device, the one or more predicted audio sources. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the neural network comprises: an audio separation network to generate one or more estimated audio sources; and an audio embedding network to generate one or more audio embeddings based on the one or more estimated audio sources, wherein the one or more audio embeddings comprise a representation of audio features. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, further comprising: training the neural network to receive a particular audio waveform associated with a particular plurality of video frames and predict one or more particular audio sources in the particular audio waveform. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the training further comprises: training the neural network to receive the particular audio waveform and predict a version of the particular audio waveform comprising particular audio sources that correspond to particular objects in the particular plurality of video frames. ### Claim 15 (depends on 13) The computer-implemented method of claim 13, wherein the training of the neural network comprises training a classifier based on active combinations cross entropy. ### Claim 16 (depends on 13) The computer-implemented method of claim 13, wherein the training of the neural network is performed at the computing device. ### Claim 17 (depends on 13) The computer-implemented method of claim 13, wherein the training of the neural network comprises unsupervised mixture invariant training. ### Claim 18 A computing device, comprising: one or more processors; and data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out operations comprising: receiving, by the computing device, an audio waveform associated with a plurality of video frames of video content; determining, by a time-domain convolutional masking network of a neural network and from the audio waveform, a neural representation comprising one or more audio frames, wherein each audio frame of the one or more audio frames comprises a respective plurality of coefficients, and wherein the respective plurality of coefficients represent one or more audio features in an encoded mixture of the audio waveform; predicting, by the neural network and based on the neural representation, one or more audio sources associated with the plurality of video frames; modifying the one or more predicted audio sources by applying a mixture consistency projection that constrains the one or more predicted audio sources to add up to the received audio waveform; and providing, by the computing device, the modified one or more predicted audio sources. ### Claim 19 An article of manufacture comprising one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out operations comprising: receiving, by the computing device, an audio waveform associated with a plurality of video frames of video content; determining, by a time-domain convolutional masking network of a neural network and from the audio waveform, a neural representation comprising one or more audio frames, wherein each audio frame of the one or more audio frames comprises a respective plurality of coefficients, and wherein the respective plurality of coefficients represent one or more audio features in an encoded mixture of the audio waveform; predicting, by the neural network and based on the neural representation, one or more audio sources associated with the plurality of video frames; modifying the one or more predicted audio sources by applying a mixture consistency projection that constrains the one or more predicted audio sources to add up to the received audio waveform; and providing, by the computing device, the modified one or more predicted audio sources. ### Claim 20 (depends on 1) The computer-implemented method of claim 1, further comprising: determining whether a given predicted audio source corresponds to an on-screen object, or is an off-screen sound.