# US 12,010,505 B2 · Low latency, low power multi-channel audio processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,505 B2
- Document ID: US12010505B2
- Status: Granted
- Kind: B2
- Application number: 17/743,045
- Filed: 2022-05-12
- Published: 2022-11-24
- Granted: 2024-06-11
- Earliest claimed: 2021-05-19
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Ashwani Arya; Tejas Bahulkar; Daniel Harris; Daniel C. Wiggins
- Web page: https://patentgrep.com/patent/US12010505B2

## CPC

- H04S 7/00: Indicating arrangements; Control arrangements, e.g. balance control
- H04S 7/304: {For headphones}
- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)}
- H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- H04S: STEREOPHONIC SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

An electronic eyewear device includes a display and a speaker system adapted to present augment reality objects and associated sounds in a scene being viewed by the user. A processor receives one or more audio tracks respectively associated with one or more augmented reality objects, encodes the audio tracks into an aggregated audio track including the audio tracks, a header for each audio track that uniquely identifies each respective audio track, and an aggregate header that identifies the number of tracks in the aggregated audio track. The processor transfers the aggregated audio track to an audio processor that uses the header for each audio track and the aggregate header to separate the audio tracks from the aggregated audio track. The audio processor processes the audio tracks independently in parallel and provides the audio tracks to the speaker system for presentation with the augmented reality objects.

## Claims

### Claim 1

An electronic eyewear device configured to present augmented reality objects to a user, comprising:
a display that presents augment reality objects in a scene being viewed by the user;
a speaker system including a first speaker for presenting a left audio signal to a left ear of a head of the user and a second speaker for presenting a right audio signal to a right ear of the head of the user;
a memory storing instructions; and
an audio processing system including a processor and an audio processor that together process the instructions to present audio associated with the augmented reality objects to the speaker system, wherein:
the processor receives one or more audio tracks respectively associated with one or more augmented reality objects, encodes the one or more audio tracks into an aggregated audio track including the one or more audio tracks, a header for each audio track that uniquely identifies each respective audio track, and an aggregate header that identifies the number of audio tracks in the aggregated audio track, and transfers the aggregated audio track to the audio processor in a first data transmission channel; and
the audio processor uses the header for each audio track and the aggregate header to separate the one or more audio tracks from the aggregated audio track, processes the one or more audio tracks independently in parallel, and provides the one or more audio tracks to the speaker system for presentation with the augmented reality objects.

### Claim 2 (depends on 1)

The electronic eyewear device of claim 1, wherein the processor further receives spatial parameter metadata relating to at least one of the one or more audio tracks, aggregates the spatial parameter metadata into aggregated spatial parameter metadata, and transfers the aggregated spatial parameter metadata to the audio processor asynchronously with respect to the aggregated audio track in a second data transmission channel.

### Claim 3 (depends on 2)

The electronic eyewear device of claim 2, wherein the audio processor comprises a head related transfer function processing module that separates the spatial parameter metadata corresponding to respective audio tracks from the aggregated spatial parameter data and processes the one or more audio tracks and spatial parameter metadata associated with the one or more audio tracks to produce the left audio signal and the right audio signal, wherein the left audio signal and the right audio signal present sounds associated with spatial positions of the augmented reality objects in the scene.

### Claim 4 (depends on 1)

The electronic eyewear device of claim 1, wherein the processor transfers the aggregated audio track to the audio processor using a single remote procedure call.

### Claim 5 (depends on 1)

The electronic eyewear device of claim 1, wherein the header for a respective audio track further comprises an indication of a version of the header and a number of samples in a frame for the respective audio track.

### Claim 6 (depends on 1)

The electronic eyewear device of claim 1, wherein the header for a respective audio track further comprises at least one of head related transfer function parameters or per-track tuning parameters for the respective audio track.

### Claim 7 (depends on 1)

The electronic eyewear device of claim 1, wherein at least one of the one or more audio tracks is compressed, and the header for each respective compressed audio track further comprises an indication of the type of compression that has been applied to the one or more compressed audio tracks.

### Claim 8 (depends on 1)

The electronic eyewear device of claim 1, wherein the processor receives one or more audio tracks respectively associated with one or more augmented reality objects at a sampling interval or at a scheduled delivery time and the header for the one or more audio tracks further comprises sampling interval data or scheduling data.

### Claim 9

A method for presenting sounds associated with augmented reality objects presented to a user in a scene, comprising:
a processor receiving one or more audio tracks respectively associated with one or more augmented reality objects;
the processor encoding the one or more audio tracks into an aggregated audio track including the one or more audio tracks, a header for each audio track that uniquely identifies each respective audio track, and an aggregate header that identifies the number of audio tracks in the aggregated audio track;
the processor transferring the aggregated audio track to an audio processor in a first data transmission channel;
the audio processor using the header for each audio track and the aggregate header to separate the one or more audio tracks from the aggregated audio track;
the audio processor processing the one or more audio tracks independently in parallel; and
the audio processor providing the one or more audio tracks to a speaker system for presentation with the augmented reality objects.

### Claim 10 (depends on 9)

The method of claim 9, further comprising the processor receiving spatial parameter metadata relating to at least one of the one or more audio tracks, aggregating the spatial parameter metadata into aggregated spatial parameter metadata, and transferring the aggregated spatial parameter metadata to the audio processor asynchronously with respect to the aggregated audio track in a second data transmission channel.

### Claim 11 (depends on 10)

The method of claim 10, wherein the audio processor comprises a head related transfer function processing module, further comprising the head related transfer function module separating the spatial parameter metadata corresponding to respective audio tracks from the aggregated spatial parameter data and processing the one or more audio tracks and spatial parameter metadata associated with the one or more audio tracks to produce a left audio signal and a right audio signal, wherein the left audio signal and the right audio signal present sounds associated with spatial positions of the augmented reality objects in the scene.

### Claim 12 (depends on 9)

The method of claim 9, further comprising the processor transferring the aggregated audio track to the audio processor using a single remote procedure call.

### Claim 13 (depends on 9)

The method of claim 9, further comprising providing an indication of a version of the header and a number of samples in a frame for a respective audio track in the header for the respective audio track.

### Claim 14 (depends on 9)

The method of claim 9, further comprising providing at least one of head related transfer function parameters or per-track tuning parameters for a respective audio track in the header for the respective audio track.

### Claim 15 (depends on 9)

The method of claim 9, wherein at least one of the one or more audio tracks is compressed, further comprising providing an indication of a type of compression that has been applied to the at least one compressed audio track in the header for the at least one compressed audio track.

### Claim 16 (depends on 9)

The method of claim 9, further comprising the processor receiving one or more audio tracks respectively associated with one or more augmented reality objects at a sampling interval or at a scheduled delivery time and providing sampling interval data or scheduling data in the header for the one or more audio tracks.

### Claim 17

A non-transitory computer readable medium including instructions for presenting sounds associated with augmented reality objects presented to a user in a scene, wherein the instructions, when performed by a processor and audio processor, configure the processor and audio processor to:
receive, by the processor, one or more audio tracks respectively associated with one or more augmented reality objects;
encode, by the processor, the one or more audio tracks into an aggregated audio track including the one or more audio tracks, a header for each audio track that uniquely identifies each respective audio track, and an aggregate header that identifies the number of audio tracks in the aggregated audio track;
transfer, by the processor, the aggregated audio track to the audio processor in a first data transmission channel;
separate, by the audio processor, the one or more audio tracks from the aggregated audio track using the header for each audio track and the aggregate header;
process, by the audio processor, the one or more audio tracks independently in parallel; and
provide, by the audio processor, the one or more audio tracks to a speaker system for presentation with the augmented reality objects.

### Claim 18 (depends on 17)

The medium of claim 17, further comprising instructions that when executed by the processor causes the processor to receive spatial parameter metadata relating to at least one of the one or more audio tracks, aggregate the spatial parameter metadata into aggregated spatial parameter metadata, and transfer the aggregated spatial parameter metadata to the audio processor asynchronously with respect to the aggregated audio track in a second data transmission channel.

### Claim 19 (depends on 18)

The medium of claim 18, further comprising instructions that when executed by the audio processor causes a head related transfer function processing module of the audio processor to separate the spatial parameter metadata corresponding to respective audio tracks from the aggregated spatial parameter data and to process the one or more audio tracks and spatial parameter metadata associated with the one or more audio tracks to produce the left audio signal and the right audio signal, wherein the left audio signal and the right audio signal present sounds associated with spatial positions of the augmented reality objects in the scene.

### Claim 20 (depends on 17)

The medium of claim 17, further comprising instructions that when executed by the processor causes the processor to transfer the aggregated audio track to the audio processor using a single remote procedure call.