# US 2026/0089455 A1 · Apparatus, method, computer program for encoding multi-microphone audio as metadata assisted spatial audio

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0089455 A1
- Document ID: US20260089455A1
- Status: Published application
- Kind: A1
- Application number: 19/214,247
- Filed: 2025-05-21
- Published: 2026-03-26
- Earliest claimed: 2024-06-04
- Assignees: Nokia Technologies Oy
- Applicants: Nokia Technologies Oy
- Inventors: Lasse Juhani LAAKSONEN; Miikka Tapani VILERMO; Arto Juhani LEHTINIEMI
- Web page: https://patentgrep.com/patent/US20260089455A1

## CPC

- H04S 7/30: {Control circuits for electronic adaptation of the sound field}
- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- H04R 1/406: {microphones}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04S 7/304: {For headphones}
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- H04S 2420/03: Application of parametric coding in stereophonic audio systems
- H04S 2420/11: Application of ambisonics in stereophonic audio systems
- H04S: STEREOPHONIC SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

An apparatus comprising means for: obtaining image-based sound source location data from image analysis of one or more captured images; encoding multi-microphone audio as metadata assisted spatial audio comprising spatial audio metadata parameters; encoding the image-based sound source location data within one or more spatial audio metadata parameters of the metadata assisted spatial audio, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data is or are one or more spatial audio metadata parameters defining a spatial distribution of audio energy.

## Claims

### Claim at position 1

1-22. (canceled)

### Claim 23

An apparatus comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
obtain image-based sound source location data from image analysis of one or more captured images;
encode multi-microphone audio as metadata assisted spatial audio comprising spatial audio metadata parameters; and
encode the image-based sound source location data within one or more spatial audio metadata parameters of the metadata assisted spatial audio, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data is or are the one or more spatial audio metadata parameters defining a spatial distribution of audio energy.

### Claim 24 (depends on 23)

An apparatus as claimed in claim 23, wherein encoding the image-based sound source location data within the one or more spatial audio metadata parameters of the metadata assisted spatial audio comprises: varying the one or more spatial audio metadata parameters, that are a result of encoding the multi-microphone audio as the metadata assisted spatial audio, based on the image-based sound source location data.

### Claim 25 (depends on 23)

An apparatus as claimed in claim 23, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data comprise at least one direction index dependent upon the image-based sound source location data.

### Claim 26 (depends on 23)

An apparatus as claimed in claim 23, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data comprise at least one coherence parameter dependent upon the image-based sound source location data.

### Claim 27 (depends on 23)

An apparatus as claimed in claim 23, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data comprise at least one spread coherence parameter dependent upon the image-based sound source location data, wherein the at least one spread coherence parameter defines coherence of a directional sound.

### Claim 28 (depends on 27)

An apparatus as claimed in claim 27, wherein the at least one spread coherence parameter is varied in dependence upon the image-based sound source location data, to include the image-based sound source location data within an increased spatial distribution of audio energy defined by a varied spread coherence parameter.

### Claim 29 (depends on 27)

An apparatus as claimed in claim 27, wherein the at least one spread coherence parameter is varied dependent upon a history of image-based sound source location data, to include a range of probable locations of the image-based sound source within the spatial distribution of audio energy defined by a varied spread coherence parameter.

### Claim 30 (depends on 23)

An apparatus as claimed in claim 23, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data comprise at least one surround coherence parameter dependent upon the image-based sound source location data, wherein the surround coherence parameter defines coherence of non-directional sound.

### Claim 31 (depends on 23)

An apparatus as claimed in claim 23, wherein the image-based sound source location data is indicative of one or more of: a width of a sound source, a size of the sound source or a direction of the sound source.

### Claim 32 (depends on 23)

An apparatus as claimed in claim 23, wherein the image analysis comprises processing one or more captured images to generate the image-based sound source location data, wherein the image-based sound source location data defines at least a location for a sound source.

### Claim 33 (depends on 23)

An apparatus as claimed in claim 23, wherein the image analysis comprises processing one or more captured images to generate the image-based sound source location data, wherein the image-based sound source location data defines at least a spatial distribution of a sound source.

### Claim 34 (depends on 23)

An apparatus as claimed in claim 23, wherein the image analysis comprises processing one or more captured images to generate the image-based sound source location data, wherein the image-based sound source location data defines at least one of a shape or a size of a sound source.

### Claim 35 (depends on 23)

An apparatus as claimed in claim 23, wherein the image analysis comprises processing one or more captured images to generate the image-based sound source location data for a sound source determined as a probable source of captured multi-microphone audio.

### Claim 36 (depends on 23)

An apparatus as claimed in claim 23, wherein the image analysis comprises processing one or more captured images to generate the image-based sound source location data, wherein the processing of the one or more captured images is constrained to a direction of a probable source of captured multi-microphone audio.

### Claim 37 (depends on 23)

An apparatus as claimed in claim 23, wherein the one or more captured images differ at least one by of time of capture or by field of view of capture.

### Claim 38 (depends on 23)

An apparatus as claimed in claim 23, wherein the apparatus is further caused to convert the one or more captured images to time-frequency tiles for image analysis.

### Claim 39 (depends on 23)

An apparatus as claimed in claim 23, wherein the apparatus is further caused to process the one or more captured images using a trained machine learning algorithm that uses synchronization of visual and audio modalities to jointly parse sounds and images, and associate parsed image regions with parsed sounds.

### Claim 40 (depends on 23)

An apparatus as claimed in claim 23, wherein the apparatus comprises a body-portable apparatus.

### Claim 41

A method comprising:
obtaining image-based sound source location data from image analysis of one or more captured images;
encoding multi-microphone audio as metadata assisted spatial audio comprising spatial audio metadata parameters; and
encoding the image-based sound source location data within one or more spatial audio metadata parameters of the metadata assisted spatial audio, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data is or are the one or more spatial audio metadata parameters defining a spatial distribution of audio energy.

### Claim 42

A non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following:
obtain image-based sound source location data from image analysis of one or more captured images;
encode multi-microphone audio as metadata assisted spatial audio comprising spatial audio metadata parameters; and
encode the image-based sound source location data within one or more spatial audio metadata parameters of the metadata assisted spatial audio, wherein the one or more spatial audio metadata parameters encoding the image-based sound source location data is or are the one or more spatial audio metadata parameters defining a spatial distribution of audio energy.
