# US 2019/0379992 A1 · Efficient rendering of virtual soundfields

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379992 A1
- Document ID: US20190379992A1
- Status: Published application
- Kind: A1
- Application number: 16/438,358
- Filed: 2019-06-11
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Brian Lloyd SCHMIDT; Samuel Charles Dicker
- Web page: https://patentgrep.com/patent/US20190379992A1

## CPC

- H04R 1/1041: {Mechanical or electronic switches, or control elements (switches in general H01H)}
- H04S 7/303: {Tracking of listener position or orientation}
- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- G10L 25/21: the extracted parameters being power information
- H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04S 7/304: {For headphones}
- H04R 2460/07: Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2400/13: Aspects of volume control, not necessarily automatic, in stereophonic sound systems
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An audio system and method of spatially rendering audio signals that uses modified virtual speaker panning is disclosed. The audio system may include a fixed number F of virtual speakers, and the modified virtual speaker panning may dynamically select and use a subset P of the fixed virtual speakers. The subset P of virtual speakers may be selected using a low energy speaker detection and culling method, a source geometry-based culling method, or both. One or more processing blocks in the decoder/virtualizer may be bypassed based on the energy level of the associated audio signal or the location of the sound source relative to the user/listener, respectively. In some embodiments, a virtual speaker that is designated as an active virtual speaker at a first time, may also be designated as an active virtual speaker at a second time to ensure the processing completes.

## Claims

### Claim 1

A method of spatially rendering audio signals, the method comprising:
modeling a virtual environment using a spatial modeler;
distributing signals from the spatial modeler across multiple virtual speakers using a spatial encoder;
representing a spatial configuration of the virtual environment using an internal spatial representation;
decoding signals from the internal spatial representation using a decoder/virtualizer;
introducing virtualized sounds into the decoded signals using a decoder/virtualizer;
selectively bypassing one or more processing blocks in the decoder/virtualizer associated with inactive virtual speakers;
combining signals from the decoder/virtualizer; and
outputting the combined signals as the audio signals.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining energy levels associated with the signals from a soundfield decoder; and
determining whether each of the detected energy levels is less than an energy threshold,
wherein the selective bypass of the one or more processing blocks includes bypassing head related transfer function (HRTF) processing of the corresponding signal from the soundfield decoder in accordance with the determination that the detected energy level of at least one of the virtual speakers is less than the energy threshold,
wherein the soundfield decoder is included in the decoder/virtualizer.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
in accordance with the determination that the detected energy level of at least one of the virtual speakers is not less than the energy threshold, performing HRTF processing of the corresponding signal from the soundfield decoder.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining whether a number of sound sources is greater than or equal to a predetermined sound source threshold,
wherein the selective bypass of the one or more processing blocks includes bypassing a plurality of detectors and directly passing signals from a soundfield decoder to a plurality of HRTF blocks when the number of sound sources is greater than or equal to the predetermined sound source threshold,
wherein the plurality of detectors and the plurality of HRTF blocks are included in the decoder/virtualizer.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
in accordance with the determination that the number of sound sources is not greater than or equal to the predetermined sound source threshold, directly passing signals from the soundfield decoder to the plurality of detectors.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining a location of each sound source; and
determining which of the multiple virtual speakers are located close to the respective sound source.

### Claim 7 (depends on 6)

The method of claim 6, wherein the determination of which of the multiple virtual speakers are located close to the respective sound source is performed at every video frame.

### Claim 8 (depends on 6)

The method of claim 6, wherein the selective bypass of the one or more processing blocks in the decoder/virtualizer includes bypassing all of the one or more processing blocks in the decoder/virtualizer associated with at least one speaker not located close to the respective sound source.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
introducing representations of movements associated with the audio signals using a rotated/translated representation; and
determining whether an amplitude of signals from the rotated/translated representation is greater than or equal to a predetermined amplitude threshold,
wherein the selective bypass of the one or more processing blocks in the decoder/virtualizer includes bypassing a soundfield decoder and a plurality of HRTF blocks when the amplitude of the signals from the rotated/translated representation is not greater than or equal to the predetermined amplitude threshold,
wherein the soundfield decoder and the plurality of HRTF blocks are included in the decoder/virtualizer.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
in accordance with the determination that the amplitude of the signals from the rotated/translated representation is greater than or equal to the predetermined amplitude threshold:
decoding signals from the rotated/translated representation, and
determining and applying head related transfer functions (HRTFs) to the decoded signals.

### Claim 11 (depends on 1)

The method of claim 1, wherein the multiple virtual speakers include the inactive virtual speakers and active virtual speakers at a first time, wherein at least one of the active virtual speakers at the first time is designated as inactive at a second time while signals are being processed.

### Claim 12

A system comprising:
a wearable head device configured to provide audio signals to a user; and
circuitry configured to spatially render the audio signals, wherein the circuitry includes:
a spatial modeler configured to model a virtual environment;
a spatial encoder configured to distribute signals from the spatial modeler across multiple virtual speakers;
an internal spatial representation configured to represent a spatial configuration of the virtual environment; and
a decoder/virtualizer configured to decode signals from the internal spatial representation and introduce virtualized sounds into the decoded signals, wherein the decoder/virtualizer includes:
a rotated/translated representation configured to introduce representations of movements associated with the audio signals;
a soundfield decoder can be configured to decode signals from the rotated/translated representation;
a plurality of head related transfer function (HRTF) blocks configured to determine and apply corresponding HRTFs to its input signals; and
a plurality of combiners configured to combine signals from the plurality of HRTF blocks and output the audio signals.

### Claim 13 (depends on 12)

The system of claim 12, further comprising:
a plurality of detectors configured to receive signals from the soundfield decoder and determine energy levels associated with the signals from the soundfield decoder; and
a plurality of first switches configured to pass the signals from the soundfield decoder to the plurality of HRTF blocks when the determined energy levels is not less than an energy threshold.

### Claim 14 (depends on 13)

The system of claim 13, further comprising:
a second switch configured to:
receive the signals from the soundfield decoder, and
selectively pass the signals from the soundfield decoder directly to the plurality of detectors or the plurality of HRTF blocks.

### Claim 15 (depends on 12)

The system of claim 12, further comprising:
a soundfield decode determination configured to:
determine whether an amplitude of a signal from the rotated/translated representation is greater than a predetermined amplitude threshold, and
in accordance with the determination that the amplitude of the signal from the rotated/translated representation is greater than the predetermined amplitude threshold, pass the signal from the rotated translated representation to the soundfield decoder.