# US 12,010,490 B1 · Audio renderer based on audiovisual information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,490 B1
- Document ID: US12010490B1
- Status: Granted
- Kind: B1
- Application number: 18/149,659
- Filed: 2023-01-03
- Granted: 2024-06-11
- Earliest claimed: 2020-08-19
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Symeon Delikaris Manias; Mehrez Souden; Ante Jukic; Matthew S. Connolly; Sabine Webel; Ronald J. Guglielmone, Jr.
- Web page: https://patentgrep.com/patent/US12010490B1

## CPC

- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04R 3/12: for distributing signals to two or more loudspeakers {(specially adapted for hearing aids H04R25/407)}
- H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)}
- H04S 7/303: {Tracking of listener position or orientation}
- H04R 1/403: {loud-speakers}
- H04R 1/406: {microphones}
- H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
- H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
- H04S 7/304: {For headphones}
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/11: Application of ambisonics in stereophonic audio systems
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

An audio renderer can have a machine learning model that jointly processes audio and visual information of an audiovisual recording. The audio renderer can generate output audio channels. Sounds captured in the audiovisual recording and present in the output audio channels are spatially mapped based on the joint processing of the audio and visual information by the machine learning model. Other aspects are described.

## Claims

### Claim 1

A method comprising:
receiving a visual feature associated with a video signal and an audio feature associated with a sound captured in a microphone signal;
receiving metadata comprising a target scene that includes a visual representation of the sound; and
determining, as output of a machine learning (ML) model using 1) the visual feature, 2) the audio feature, and 3) the target scene as input, one or more mapping parameters that, when applied to the microphone signal, generates one or more output audio channels that includes the sound of the microphone signal that is spatially mapped according to a location of the visual representation within the target scene and the mapping parameters are determined based on one or more correlations between the sound and the visual feature.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more mapping parameters includes a direction-of-arrival (DoA) estimation of the sound.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more mapping parameters includes the location of the visual representation of the sound within the target scene.

### Claim 4 (depends on 1)

The method of claim 1, wherein the target scene is visually the same as a record scene represented by the video signal.

### Claim 5 (depends on 1)

The method of claim 1, wherein the visual representation is a virtual object associated with a recorded object that is a sound source of the sound within a recorded scene of the video signal, wherein the location of the virtual object within the target scene is different than a location of the recorded object within the recorded scene.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more output audio channels are associated with a target output audio format that is one of: a binaural audio format comprising a left audio channel and a right audio channel, a channel-based loudspeaker format, and a spherical surround sound format.

### Claim 7 (depends on 1)

The method of claim 1, wherein the visual representation is an avatar of a person within a recorded scene represented by the video signal, wherein the person is a sound source for the sound.

### Claim 8

A non-transitory machine-readable medium storing instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:
receive a visual feature associated with a video signal and an audio feature associated with a sound captured in a microphone signal;
receive metadata comprising a target scene that includes a visual representation of the sound; and
determine, as output of a machine learning (ML) model using 1) the visual feature, 2) the audio feature, and 3) the target scene as input, one or more mapping parameters that, when applied to the microphone signal, generates one or more output audio channels that includes the sound of the microphone signal that is spatially mapped according to a location of the visual representation within the target scene and the mapping parameters are determined based on one or more correlations between the and the visual feature.

### Claim 9 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the one or more mapping parameters includes a direction-of-arrival (DoA) estimation of the sound.

### Claim 10 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the one or more mapping parameters includes the location of the visual representation of the sound within the target scene.

### Claim 11 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the target scene is visually the same as a record scene represented by the video signal.

### Claim 12 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the visual representation is a virtual object associated with a recorded object that is a sound source of the sound within a recorded scene of the video signal, wherein the location of the virtual object within the target scene is different than a location of the recorded object within the recorded scene.

### Claim 13 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the one or more output audio channels are associated with a target output audio format that is one of: a binaural audio format comprising a left audio channel and a right audio channel, a channel-based loudspeaker format, and a spherical surround sound format.

### Claim 14

An electronic device comprising:
at least one processor; and
memory having instructions stored therein which when executed by the at least one processor causes the electronic device to:
receive input audio data that includes a sound, video data, and metadata comprising a target scene that includes a visual representation of the sound; and
generate output audio data in a target output audio format as output of a machine learning (ML) model using 1) the input audio data, 2) the video data, and 3) the target scene as input, wherein the ML model maps the input audio data to the output audio data according to the target output audio format, wherein the output audio data comprises the sound that is spatially mapped according to a location of the visual representation within the target scene, wherein the ML model outputs the output audio data based on one or more correlations between the sound and visual information of the video data.

### Claim 15 (depends on 14)

The electronic device of claim 14, wherein the target output audio format is defined in the metadata.

### Claim 16 (depends on 14)

The electronic device of claim 14, wherein the target output audio format comprises at least one of: a binaural audio format, a channel-based loudspeaker format, and a spherical surround sound format.

### Claim 17 (depends on 14)

The electronic device of claim 14, wherein the target output audio format is a first audio format, wherein the input audio data is in a second audio format that is different than the first audio format.

### Claim 18 (depends on 14)

The electronic device of claim 14, wherein the target scene is visually the same as a scene represented by the video data.

### Claim 19 (depends on 14)

The electronic device of claim 14, wherein the visual representation is a virtual object associated with a recorded object that is a sound source of the sound within a recorded scene represented by the video data, wherein the location of the virtual object within the target scene is different than a location of the recorded object within the recorded scene.

### Claim 20 (depends on 14)

The electronic device of claim 14, wherein the output audio data is generated based on mapping the sound of the input audio data to the location of the visual representation within the target scene that is defined in the metadata.