Back to results
US 2020/0349975 A1Published application
Video tagging by correlating visual features to sound tags
AssigneeSony Interactive Entertainment Inc.
InventorsSudha Krishnamurthy, Xiaoyu Liu
Life of this patent
Estimating the term…
Abstract
Automatically recommending sound effects based on visual scenes enables sound engineers during video production of computer simulations, such as movies and video games. This recommendation engine may be accomplished by classifying SFX and using a machine learning engine to output a first of the classified SFX for a first computer simulation based on learned correlations between video attributes of the first computer simulation and the classified SFX.
Claims20
An apparatus, comprising:
- at least one processor;
- and
- at least one computer storage that is not a transitory signal and that comprises instructions executable by the at least one processor to:
- classify sound effects (SFX) to render classified SFX;
- use at least one machine learning engine to output at least a first of the classified SFX for at least a first computer simulation at least in part based on learned correlations between video attributes of the first computer simulation and the classified SFX;
- input the first computer simulation without sound to at least a first neural network (NN) trained to learn correlations between visual features in video and SFX tags;
- and
- input to the first NN information from at least a first noisy SFX model comprising ground truth classifications of noisy SFX.
5 more elements
(canceled)
An apparatus, comprising:
- at least one processor;
- and
- at least one computer storage that is not a transitory signal and that comprises instructions executable by the at least one processor to:
- train at least a first sound effect (SFX) recommendation engine at least in part by:
- inputting silent video frames and noisy SFX labels to plural residual neural networks (Resnet);
- inputting an output of the Resnet to at least one bi-directional gated recurrent unit to render a vector;
- recommend at least one SFX for at least a first video with no sound at least in part by:
- inputting an output of the Resnet to at least one trained model also configured for receiving as input at least a second video without sound to output at least one SFX tag representing a recommended SFX for the second video.
5 more elements
A method comprising:
- classifying first and second sound effects in a first video or a first computer simulation;
- and
- based at least in part on the classifying, providing sound effect predictions for a second video or second computer simulation, wherein the classifying comprises:
- inputting the first computer simulation without sound to at least a first neural network (NN) trained to learn correlations between visual features in video and sound effect (SFX) tags;
- inputting to the first NN information from at least a first noisy SFX model comprising ground truth classifications of noisy SFX and/or predicted SFX tags.
2 more elements
(canceled)
Dependencies are inferred from the claim text.