# US 2020/0288259 A1 · Spatial audio upmixing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288259 A1 - Document ID: US20200288259A1 - Status: Published application - Kind: A1 - Application number: 16/645,433 - Filed: 2018-09-28 - Published: 2020-09-10 - Earliest claimed: 2017-09-29 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Stephen E. Pinto; Christopher T. Eubank; Matthew S. Connolly - Web page: https://patentgrep.com/patent/US20200288259A1 ## CPC - H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)} - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)} - H04S 7/304: {For headphones} - H04S 1/007: {in which the audio signals are in digital form (data reduction aspects thereof based on psychoacoustics G10L19/02)} - H04S 7/305: {Electronic adaptation of stereophonic audio signals to reverberation of the listening space (H04S7/301 takes precedence)} - H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S 2400/13: Aspects of volume control, not necessarily automatic, in stereophonic sound systems - H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction - H04S: STEREOPHONIC SYSTEMS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Spatial audio upmixing enables mixing spatial sound on a more granular per-channel level than is possible using conventional upmixing. A spatial bed is a multi-channel audio content that represents a complete sound field description, e.g., a virtual sphere of sound, for example surrounding a simulated reality listener in a simulated reality environment. A new spatial bed is generated by combining sections of at least two of such spatial beds. Other embodiments are also described and claimed. ## Claims ### Claim 1 A computer-implemented method for creating a custom mix of different spatial audio objects, the method comprising: receiving a selection of a first input spatial audio object; receiving a selection of a first portion of a custom mix visualization where sound of the first input spatial audio object is to be rendered; receiving a selection of a second input spatial audio object; receiving a selection of a second portion of the custom mix visualization where sound of the second input spatial audio object is to be rendered; and generating a new spatial audio object that represents an array of virtual sound sources that define a sound field of the new spatial audio object, wherein sound of the first input object forms a part of the array of virtual sources, and sound of the second input object forms another part of the array of virtual sources. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the custom mix visualization is a globe, and wherein the array of virtual sound sources is a spherical array of virtual sound sources. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the selected first portion has an irregular spherical shape. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein the selected first portion is an outer surface of a spherical wedge. ### Claim 5 (depends on 2) The computer-implemented method of claim 2, wherein the selected first portion is an outer surface of a spherical sector. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising receiving a selection of a third input spatial audio object; and receiving a selection of a third portion of the custom mix visualization where sound of the third input object is to be rendered, wherein the third portion overlaps the first portion, and wherein generating the new spatial audio object comprises combining sound of the third input object with sound of the first input object to be rendered at the third portion of the custom mix visualization. ### Claim 7 (depends on 1) The computer implemented method of claim 1 further comprising receiving a change sound command to change a sound of a target portion of the custom mix visualization, wherein generating the new spatial audio object comprises defining a filter or a gain in accordance with the change sound command that is to be applied to the sound of any input object that is to be rendered at the target portion. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: upsampling a multichannel audio recording onto a uniformly spaced spherical grid that corresponds to a spherical array of virtual sound sources, to generate one of the first or second input spatial audio objects. ### Claim 9 (depends on 1) The computer-implemented method of claim 1 further comprising rendering the new spatial audio object for output as spatial sound by a speaker system. ### Claim 10 (depends on 1) The computer-implemented method of claim 1 further comprising binaural rendering the new spatial audio object for output as spatial sound through headphones. ### Claim 11 (depends on 1) The computer-implemented method of claim 1 further comprising generating a visualization of each of the first and second input spatial audio objects as a globe whose outer skin has an image representative of a recorded sound scene captured in the input spatial audio object. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein each of the first and second input spatial audio object comprises multichannel audio content that has been projected onto an N-element spherical grid array of virtual sound sources that define a sound field surrounding a central listening position of the respective spatial audio object, the new spatial audio object comprises an N-element spherical grid array of virtual sound sources, and generating the new spatial audio object comprises performing a matrix sum of the virtual sound sources in the first and second input audio objects. ### Claim 13 A computer system for creating a custom mix of different of spatial audio objects, the computer system comprising a processor and memory having stored therein instructions that when executed by the processor: receive a selection of a first input spatial audio object; receive a selection of a first portion of a custom mix visualization where sound of the first input spatial audio object is to be rendered; receive a selection of a second input spatial audio object; receive a selection of a second portion of the custom mix visualization where sound of the second input spatial audio object is to be rendered; and generate a new spatial audio object that represents an array of virtual sound sources that define a sound field of the new spatial audio object, wherein sound of the first input object forms a part of the array of virtual sources, and sound of the second input object forms another part of the array of virtual sources. ### Claim 14 (depends on 13) The computer system of claim 13, wherein the custom mix visualization is a globe, and wherein the array of virtual sound sources is a spherical array of virtual sound sources. ### Claim 15 (depends on 14) The computer system of claim 14, wherein the selected first portion has an irregular spherical shape. ### Claim 16 (depends on 13) The computer system of claim 13 wherein the memory has further instructions stored therein that when executed by the processor: receive a selection of a third input spatial audio object; and receive a selection of a third portion of the custom mix visualization where sound of the third input object is to be rendered, wherein the third portion overlaps the first portion, and wherein generating the new spatial audio object comprises combining sound of the third input object with sound of the first input object to be rendered at the third portion of the custom mix visualization. ### Claim 17 (depends on 13) The computer system of claim 13 wherein the memory has further instructions stored therein that when executed by the processor: receive a change sound command to change a sound of a target portion of the custom mix visualization, wherein generating the new spatial audio object comprises defining a filter or a gain in accordance with the change sound command that is to be applied to the sound of an input object that is to be rendered at the target portion. ### Claim 18 (depends on 13) The computer system of claim 13 wherein the memory has stored therein further instructions that when executed by the processor upsample a multichannel audio recording onto a uniformly spaced spherical grid that corresponds to a spherical array of virtual sound sources, to generate one of the first or second input spatial audio objects. ### Claim 19 (depends on 13) The computer system of claim 13 wherein the memory has stored therein further instructions that when executed by the processor render the new spatial audio object for output as spatial sound by a speaker system. ### Claim 20 (depends on 13) The computer system of claim 13 wherein the memory has stored therein further instructions that when executed by the processor perform binaural rendering of the new spatial audio object for output as spatial sound through headphones. ### Claim 21 (depends on 13) The computer system of claim 13 wherein the memory has stored therein further instructions that when executed by the processor generate a visualization of each of the first and second input spatial audio objects as a globe whose outer skin has an image representative of a recorded sound scene captured in the input spatial audio object. ### Claim 22 (depends on 13) The computer system of claim 13 wherein each of the first and second input spatial audio object comprises multichannel audio content that has been projected onto an N-element spherical grid array of virtual sound sources that define a sound field surrounding a central listening position of the respective spatial audio object, the new spatial audio object comprises an N-element spherical grid array of virtual sound sources, and generating the new spatial audio object comprises performing a matrix sum of the virtual sound sources in the first and second input audio objects.