Self-supervised AI-assisted sound effect recommendation for silent video
AssigneeSony Interactive Entertainment Inc.
InventorSudha Krishnamurthy
Life of this patent
Estimating the term…
Abstract
Sound effect recommendations for visual input are generated by training machine learning models that learn coarse-grained and fine-grained audio-visual correlations from a reference image, a positive audio signals, and a negative audio signal. A positive audio embedding related to the reference image is generated from the positive audio signal and a negative audio embedding is generated from a negative audio signal. A machine learning algorithm uses the reference image, the positive audio embedding and the negative audio embedding as inputs to train a visual-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image.
Claims20
A method for training a Sound Effect Recommendation Network, comprising:
- a) generating a positive audio embedding from a positive audio signal wherein the positive audio signal is related to a reference image wherein the reference image is a visual input in the form of a still image or part of a video sequence;
- b) generating a negative audio embedding from a negative audio signal;
- c) using a machine learning algorithm with the reference image, the positive audio embedding and the negative audio embedding as inputs to train a visual-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image;
- and
- d) determining the positive audio signal from a noisy positive audio signal using a trained similarity neural network with an audio database and the noisy positive audio signal to select the positive audio signal from the audio database.
A system for training a Sound Effect Recommendation Network, comprising:
- a Processor;
- a Memory coupled to the Processor;
- Non-transitory instructions embedded in the memory that when executed cause the processor to carry out the method for training a sound effect recommendation network comprising;
- a) generating a positive audio embedding from a positive audio signal wherein the positive audio signal is related to a reference image wherein the reference image is a visual input in the form of a still image or part of a video sequence;
- b) generating a negative audio embedding from a negative audio signal;
- c) using a machine learning algorithm with the reference image, the positive audio embedding and the negative audio embedding as inputs to train an image-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image;
- and
- d) determining the positive audio signal from a noisy positive audio signal using a trained similarity neural network with an audio database and the noisy positive audio signal to select the positive audio signal from the audio database.
Non-transitory instructions embedded in a computer readable medium that when executed by a computer cause the computer to carry out the method for training a Sound Recommendation Network comprising:
- a) generating a positive audio embedding from a positive audio signal wherein the positive audio signal is related to a reference image wherein the reference image is a visual input in the form of a still image or part of a video sequence;
- b) generating a negative audio embedding from a negative audio signal;
- c) using a machine learning algorithm with the reference image, the positive audio embedding and a negative audio embedding as inputs to train a visual-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image;
- and
- d) determining the positive audio signal from a noisy positive audio signal using a trained similarity neural network with an audio database and the noisy positive audio signal to select the positive audio signal from the audio database.
Dependencies are inferred from the claim text.