Audio renderer based on audiovisual information
AssigneeApple Inc.
InventorsSymeon Delikaris Manias, Mehrez Souden, Ante Jukic, Matthew S. Connolly, Sabine Webel, Ronald J. Guglielmone, Jr.
Life of this patent
Estimating the term…
Abstract
An audio renderer can have a machine learning model that jointly processes audio and visual information of an audiovisual recording. The audio renderer can generate output audio channels. Sounds captured in the audiovisual recording and present in the output audio channels are spatially mapped based on the joint processing of the audio and visual information by the machine learning model. Other aspects are described.
Claims20
A method comprising:
- receiving a visual feature associated with a video signal and an audio feature associated with a sound captured in a microphone signal;
- receiving metadata comprising a target scene that includes a visual representation of the sound;
- and
- determining, as output of a machine learning (ML) model using 1) the visual feature, 2) the audio feature, and 3) the target scene as input, one or more mapping parameters that, when applied to the microphone signal, generates one or more output audio channels that includes the sound of the microphone signal that is spatially mapped according to a location of the visual representation within the target scene and the mapping parameters are determined based on one or more correlations between the sound and the visual feature.
A non-transitory machine-readable medium storing instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:
- receive a visual feature associated with a video signal and an audio feature associated with a sound captured in a microphone signal;
- receive metadata comprising a target scene that includes a visual representation of the sound;
- and
- determine, as output of a machine learning (ML) model using 1) the visual feature, 2) the audio feature, and 3) the target scene as input, one or more mapping parameters that, when applied to the microphone signal, generates one or more output audio channels that includes the sound of the microphone signal that is spatially mapped according to a location of the visual representation within the target scene and the mapping parameters are determined based on one or more correlations between the and the visual feature.
An electronic device comprising:
- at least one processor;
- and
- memory having instructions stored therein which when executed by the at least one processor causes the electronic device to:
- receive input audio data that includes a sound, video data, and metadata comprising a target scene that includes a visual representation of the sound;
- and
- generate output audio data in a target output audio format as output of a machine learning (ML) model using 1) the input audio data, 2) the video data, and 3) the target scene as input, wherein the ML model maps the input audio data to the output audio data according to the target output audio format, wherein the output audio data comprises the sound that is spatially mapped according to a location of the visual representation within the target scene, wherein the ML model outputs the output audio data based on one or more correlations between the sound and visual information of the video data.
Dependencies are inferred from the claim text.