# US 2020/0288258 A1 · File format for spatial audio > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288258 A1 - Document ID: US20200288258A1 - Status: Published application - Kind: A1 - Application number: 16/645,422 - Filed: 2018-09-25 - Published: 2020-09-10 - Earliest claimed: 2017-09-29 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Matthew S. Connolly; Christopher T. Eubank; Stephen E. Pinto; Tomlinson Holman - Web page: https://patentgrep.com/patent/US20200288258A1 ## CPC - H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)} - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing - G10L 19/167: {Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes} - H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction - H04S 2420/11: Application of ambisonics in stereophonic audio systems - H04S: STEREOPHONIC SYSTEMS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract An audio asset library containing audio assets formatted in accordance with a file format for spatial audio includes asset metadata that enables simulated reality (SR) application developers to compose sounds for use in SR applications. The audio assets are formatted to include audio data encoding a sound capable of being composed into a SR application along with asset metadata describing not only how the sound was encoded, but also how a listener in SR environment experiences the sound. A SR developer platform is configured so that developers can compose sound for SR objects using audio assets stored in the audio library, including editing the asset metadata to include transformation parameters that support dynamic transformation of the asset metadata in the SR environment to alter how the SR listener experiences the composed sound. Other embodiments are also described and claimed. ## Claims ### Claim 1 A computer-implemented method for creating an audio asset library of sounds for use in a simulated reality (SR) application, the method comprising: receiving audio data that has encoded therein a sound; receiving metadata describing how the sound is encoded; creating an audio asset containing asset metadata and the audio data, the asset metadata including i) the received metadata and metadata describing how the sound is to be rendered as spatial audio through a SR application; and storing the audio asset in an audio asset library. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein how the sound is to be rendered comprises how spatial sound is simulated in a sound playback system. ### Claim 3 (depends on 1) The computer-implemented method of claim 1 further comprising editing the asset metadata to include one or more parameters that enable dynamic transformation of the asset metadata in the SR application, wherein the dynamic transformation is to be performed during playback or experiencing of the sound by applying the one or more parameters to alter how the audio asset is rendered through the SR application. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the one or more parameters comprise a relationship parameter for dynamically transforming how the audio asset is to be rendered through the SR application based on a relationship between a listener and a SR object in a SR environment provided by the SR application and to which the audio asset is attached, the relationship including any one of a location and orientation of the listener relative to the SR object, the location and orientation affecting the rendering of the audio asset in the SR application. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the relationship between the listener and the SR object as determined in the SR environment is used to set the relationship parameter for dynamically transforming how the audio asset is rendered in the SR application. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the audio data having encoded therein the sound comprises any one or more of: a monaural recording of an individual sound; a multi-channel recording of a sound environment, including any one of a recording produced using an array of microphones, a recorded mix of multiple sound sources including a mix of multiple discrete sounds, and a recording that preserves spatial characteristics of recorded sound; and synthesized audio data for producing one or more sounds. ### Claim 7 (depends on 1) The method of claim 1, wherein the asset metadata contained in the audio asset comprises: global metadata for the audio asset; and channel metadata for any one or more channels of the audio asset. ### Claim 8 (depends on 1) The method of claim 1, wherein the metadata describing how the sound was encoded describes one or more characteristics associated with how the sound was captured, and wherein the metadata describing how to render the audio asset comprises any one or more of: a location of the sound, including a discrete location or a location encoded using a higher order ambisonic (HOA) format; at least one directional characteristic of the sound per channel of audio data; an orientation of the sound, including an orientation per channel of audio data; an original sound pressure level (SPL) or alternatively a sound power level of the sound, including a distance at which the SPL or sound power level was captured; a size or shape of the sound expressed by a polygonal mesh or volumetric size; and a playback rate for the sound, including a global playback rate for all channels of audio data, wherein the asset metadata further includes a transformation parameter for transforming the asset metadata before rendering the audio asset for experience by the listener, including a relationship parameter representing a relationship between the listener in a SR environment provided by the SR application and a SR object to which the audio asset is attached. ### Claim 9 (depends on 8) The method of claim 8, wherein the one or more characteristics associated with how the sound was encoded comprises a description of a recording system used to capture the sound, including a description of an array of microphones. ### Claim 10 (depends on 1) The method of claim 1, further comprising: editing the audio asset to add metadata describing how the audio asset was used in the SR application, including identifying a SR object to which the audio asset was attached. ### Claim 11 A computer-implemented method for composing sound for a simulated reality (SR) environment, the method comprising: retrieving an audio asset stored in an audio library, wherein the audio asset contains audio data and asset metadata, the audio data having therein a sound encoded in accordance with or as specified in the asset metadata, the asset metadata further describing how the audio asset is to be rendered in a SR application; and composing the retrieved audio asset into the SR application, the composing including previewing how the audio asset is to be rendered in the SR application in accordance with the asset metadata, editing the audio asset to enable dynamic transformation of the asset metadata in the SR application to alter how the audio asset is rendered in the SR application, and attaching the edited audio asset to a SR object in the SR application. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein how the audio asset is rendered in the SR application includes how spatial sound is simulated in a sound playback system using binaural rendering. ### Claim 13 (depends on 11) The computer-implemented method of claim 11, wherein editing the audio asset to enable dynamic transformation of the asset metadata in the SR application includes adding one or more transformation parameters to the asset metadata, the one or more transformation parameters including a relationship parameter for dynamically transforming how the audio asset is to be rendered based on a relationship between a listener and a SR object in a SR environment provided by the SR application, wherein the audio asset is attached to the SR object, the relationship including any one of a location and orientation of the listener relative to the SR object as determined in the SR environment. ### Claim 14 (depends on 11) The computer-implemented method of claim 11 wherein editing the audio asset to enable dynamic transformation of the asset metadata in the SR application comprises adding a parameter for dynamically transforming how the audio asset is to be rendered for the SR environment provided by the SR application, wherein the dynamic transformation is based on a characteristic of a further SR object in the SR application, wherein the audio asset is attached to the SR object, the characteristic of the SR object includes any one of a location of the further object, an orientation of the further object, a velocity of the further object, or other aspect defined in the application. ### Claim 15 (depends on 11) The computer-implemented method of claim 11, further comprising storing the edited audio data and asset metadata in the audio library from which the audio asset was retrieved. ### Claim 16 (depends on 11) The computer-implemented method of claim 11 wherein the audio data includes any one or more of: a monaural recording of an individual sound; a multi-channel recording of a sound environment, including any one of a recording produced using an array of microphones, a recorded mix of multiple sound sources including a mix multiple discrete sounds, and a recording that preserves spatial characteristics of recorded sound; and synthesized audio data for producing one or more sounds. ### Claim 17 (depends on 11) The computer-implemented method of claim 11, wherein the asset metadata contained in the audio asset includes: global metadata for the audio asset; and channel metadata for any one or more channels of the audio asset. ### Claim 18 (depends on 11) The computer-implemented method of claim 11, wherein the metadata describing how the sound was encoded describes one or more characteristics associated with how the sound was captured, and wherein the metadata describing how to render the audio asset comprises one or more of: a location of the sound, including a discrete location or a location encoded using a higher order ambisonic (HOA) format; at least one directional characteristic of the sound per channel of audio data, the directional characteristic representing any one of a shape and a mathematical function used to convey directionality of the sound the shape including any one of a sphere, semi-sphere or cone, the mathematical function including any one of unit-sphere or cardioid polar response or a look up table that defines an arbitrary directivity; an orientation of the sound, including an orientation per channel of audio data; an original sound pressure level (SPL) or alternatively a sound power level of the sound, including a distance at which the SPL or sound power level was captured; a size or shape of the sound expressed as a polygonal mesh or a volumetric size; and a playback rate for the sound, including a global playback rate for all channels of audio data, wherein the asset metadata further includes a transformation parameter that is to be used for transforming the asset metadata before rendering the audio asset, including a relationship parameter representing a relationship between a listener in a SR environment provided by the SR application and a SR object to which the audio asset is attached. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein the one or more characteristics associated with how the sound was encoded in the audio data comprises a description of one or more recording systems used to capture the sound, including an array of microphones. ### Claim 20 (depends on 11) The computer-implemented method of claim 11 further comprising: editing the retrieved audio asset to add metadata describing how the audio asset was used in the SR application, including identifying a SR object to which the audio asset was attached, to produce an edited audio asset; and publishing the edited audio asset to the audio library. ### Claim 21 A computer-implemented method for processing sound in a simulated reality (SR) environment, the method comprising: configuring a SR environment in accordance with a SR application, the SR application having a SR object to which an audio asset is attached, wherein the audio asset contains audio data and asset metadata, the audio data having therein a sound encoded in accordance with the asset metadata, the asset metadata having one or more transformation parameters for describing how the audio asset is to be rendered; receiving a request to play the sound as encoded in the audio asset; determining a relationship between a listener and the SR object to which the audio asset is attached, including any one or more of a location and orientation of the listener relative to the SR object in the SR environment, as provided by the SR application, wherein the location and orientation affects how the audio asset is to be rendered in the SR application; setting one of the one or more transformation parameters of the asset metadata with the determined relationship between the listener and the SR object, wherein setting the transformation parameter dynamically transforms the asset metadata to alter how the audio asset is to be rendered through the SR application; and rendering the audio asset to a sound playback system in accordance with the SR application and in accordance with the transformed asset metadata. ### Claim 22 (depends on 21) The method of claim 21, further comprising: receiving, from the SR application, a request to edit the audio asset to add metadata describing how the audio asset was used in the SR application, including identifying the SR object to which the audio asset was attached; editing the audio asset according to the received request to produce an edited audio asset; and publishing the edited audio asset to the audio library. ### Claim 23 (depends on 21) The method of claim 21 further comprising: receiving, from the SR application, a request to edit the audio asset to add metadata describing how the audio asset was used in the SR application, including identifying the SR object to which the audio asset was attached; editing metadata of the audio asset according to the received request; and publishing the edited metadata to the audio library.