Associating Audio with Three-Dimensional Objects in Videos

A non-transitory computer readable medium storing instructions for locating and tracking one or more audio sources recorded by a plurality of microphones, wherein the instructions when executed by one or more computer processors cause the one or more processors to perform steps comprising:

• receiving at least one position for each of at least one camera;
• receiving positions for each of the plurality of microphones.
• receiving at least one video recorded by a camera;
• receiving a plurality of audio signals, each audio signal recorded by a microphone of the plurality of microphones;
• applying source separation to the plurality of audio signals to generate one or more audio source signals, each audio source signal having originated from a respective audio source of the one or more audio sources;
• estimating, for each audio source, a position associated with the audio source;
• estimating a position for each of one or more visual objects based on the at least one video;
• matching each of the one or more audio sources to a corresponding visual object of the one or more visual objects based on the estimated position of the audio source and the estimated position of the visual object;
• tracking movement of each of the one or more visual objects to generate visual object position data associated with movement of the one or more visual objects;
• and
• storing audio source position data for each of the one or more audio sources based on the visual object position data associated with the visual object to which the audio source was matched.

A method for locating and tracking one or more audio sources recorded by a plurality of microphones, the method comprising:

• receiving at least one position for each of at least one camera;
• receiving positions for each of the plurality of microphones.
• receiving at least one video recorded by a camera;
• receiving a plurality of audio signals, each audio signal recorded by a microphone of the plurality of microphones;
• estimating a position for each of one or more visual objects based on the at least one video corresponding to a first time in the at least one video;
• applying source separation to the plurality of audio signals to generate one or more audio source signals, each audio source signal having originated from a respective audio source of the one or more audio sources;
• estimating, for each audio source, a position associated with the audio source corresponding to the first time;
• matching each of the one or more audio sources to a corresponding visual object of the one or more visual objects based on the estimated position of the audio source and the estimated position of the visual object;
• tracking the position of each of the one or more audio sources by tracking the visual object to which the audio source is matched.