Video-Informed Spatial Audio Expansion
AssigneeGoogle Inc.
InventorsMarcin Gorzel, Balineedu Adsumilli
Life of this patent
Estimating the term…
Abstract
Assigning spatial information to audio segments is disclosed. A method includes receiving a first audio segment that is non-spatialized and is associated with first video frames; identifying visual objects in the first video frames; identifying auditory events in the first audio segment; identifying a match between a visual object of the visual objects and an auditory event of the auditory events; and assigning a spatial location to the auditory event based on a location of the visual object.
Claims20
A method of assigning spatial information to audio segments, comprising:
- receiving a first audio segment, wherein the first audio segment is non-spatialized, and wherein the first audio segment is associated with first video frames;
- identifying visual objects in the first video frames;
- identifying auditory events in the first audio segment;
- identifying a match between a visual object of the visual objects and an auditory event of the auditory events;
- and
- assigning a spatial location to the auditory event based on a location of the visual object.
A method of assigning spatial information to audio events in a video, comprising:
- demultiplexing the video to obtain an audio track and video frames;
- assigning respective visual labels to visual objects in the video frames;
- separating the audio track into multiple tracks;
- assigning respective audio labels to the multiple tracks;
- automatically matching some of the respective audio labels to some of the visual labels;
- and
- assigning respective spatial locations to the some of the respective audio labels based on respective locations of the some of the visual objects.
A system for assigning spatial information to audio segments, comprising:
- a memory;
- and
- a processor, the processor configured to execute instructions stored in the memory to:
- receive a first audio segment,
- wherein the first audio segment is non-spatialized, and
- wherein the first audio segment is associated with first video frames;
- identify visual objects in the first video frames;
- identify auditory events in the first audio segment;
- in response to identifying a match between a visual object of the visual objects and an auditory event of the auditory events, assign a spatial information to the auditory event;
- and
- in response to not identifying the match, assign the auditory event to a diffuse sound field.
Dependencies are inferred from the claim text.