US 9,749,738 B1 · Synthesizing audio corresponding to a virtual microphone location

Abstract

Disclosed is a system and method for generating a model of the geometric relationships between various audio sources recorded by a multi-camera system. The spatial audio scene module associates source signals, extracted from recorded audio, of audio sources to visual objects identified in videos recorded by one or more cameras. This association may be based on estimated positions of the audio sources based on relative signal gains and delays of the source signal received at each microphone. The estimated positions of audio sources are tracked indirectly by tracking the associated visual objects with computer vision. A virtual microphone module may receive a position for a virtual microphone and synthesize a signal corresponding to the virtual microphone position based on the estimated positions of the audio sources.

Claims20

Dependent claims

A method for generating a synthesized audio signal for a virtual microphone, the method comprising:

receiving at least one video;
receiving one or more visual objects identified in the at least one video;
receiving a plurality of audio signals each originating from a respective audio source, each audio source associated with one of the visual objects;
tracking a time-varying position of each of the one or more visual objects in the at least one video;
receiving a virtual microphone position;
and
generating a synthesized audio signal from a combination of the plurality of audio signals based on the virtual microphone position and the tracked time-varying positions of the one or more visual objects, the synthesized audio signal approximating an audio signal that would have been captured by a microphone positioned at the virtual microphone position during capture of the plurality of audio signals.

4 more elements

A non-transitory computer readable medium storing instructions for generating a synthesized audio signal for a virtual microphone, wherein the instructions when executed by one or more computer processors cause the one or more processors to perform steps comprising:

receiving at least one video;
receiving one or more visual objects identified in the at least one video;
receiving a plurality of audio signals each originating from a respective audio source, each audio source associated with one of the visual objects;
tracking a position of each of the one or more visual objects in the at least one video;
receiving a virtual microphone position;
and
generating a synthesized audio signal from a combination of the plurality of audio signals based on the virtual microphone position and the tracked positions of the one or more visual objects, the synthesized audio signal approximating an audio signal that would have been captured by a microphone positioned at the virtual microphone position during capture of the plurality of audio signals.

4 more elements

Dependencies are inferred from the claim text.

Synthesizing audio corresponding to a virtual microphone location

Life of this patent

Synthesizing audio corresponding to a virtual microphone location

Life of this patent

Abstract

Claims20