# US 10,321,256 B2 · Adaptive audio construction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,321,256 B2 - Document ID: US10321256B2 - Status: Granted - Kind: B2 - Application number: 15/547,043 - Filed: 2016-02-02 - Published: 2018-01-11 - Granted: 2019-06-11 - Earliest claimed: 2015-02-03 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Glenn N. Dickins; Richard J. Cartwright - Web page: https://patentgrep.com/patent/US10321256B2 ## CPC - H04S 7/303: {Tracking of listener position or orientation} - G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing - H04R 5/027: Spatial or constructional arrangements of microphones, e.g. in dummy heads - H04S 7/00: Indicating arrangements; Control arrangements, e.g. balance control - H04S 7/30: {Control circuits for electronic adaptation of the sound field} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04S 3/006: {in which a plurality of audio signals are transformed in a combination of audio signals and modulated signals, e.g. CD-4 systems} - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction - H04S: STEREOPHONIC SYSTEMS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Systems, methods, and computer program products for creating an object-based audio signal from an audio input are described. The audio input includes one or more audio channels that are recorded to collectively define an audio scene. The one or more audio channels are captured from a respective one or more spatially separated microphones disposed in a stable spatial configuration. A system receives the audio input. The system performs spatial analysis on the one or more audio channels to identify one or more audio objects within the audio scene. The system determines contextual information relating to the one or more audio objects. The system defines respective audio streams including audio data relating to at least one of the identified one or more audio objects. The system then outputs an object-based audio signal including the audio streams and the contextual information. ## Claims ### Claim 1 A method for creating an object-based audio signal from an audio input, the audio input including one or more audio channels that are recorded to collectively define an audio scene, the one or more audio channels being captured from a respective one or more spatially separated microphones disposed in a stable spatial configuration, the method including the steps of: a) receiving, by a spatial audio analysis module, the audio input; b) performing spatial analysis on the one or more audio channels to identify one or more audio objects within the audio scene; c) determining contextual information relating to the one or more audio objects, wherein the contextual information includes spatial properties of the one or more audio objects; d) defining respective audio streams including audio data relating to at least one of the identified one or more audio objects; e) outputting, by the spatial audio analysis module, an object-based audio signal including the audio streams and the contextual information; f) receiving, by an automated effects processing module, the object-based audio signal; g) performing effects processing on one or more of the audio streams to generate a modified object-based audio signal, wherein the effects processing is based on the contextual information and on external context information that is external to the audio signal, and wherein the effects processing includes performing Doppler frequency shifting for a given audio stream based on the spatial properties included in the contextual information; and h) outputting, by the automated effects processing module, the modified object-based audio signal as an encoded signal. ### Claim 2 (depends on 1) A method according to claim 1 including the further step of: receiving, by the spatial audio analysis module, the external context information, the external context information being relevant to the audio input. ### Claim 3 (depends on 2) A method according to claim 2 wherein the spatial analysis is performed based on the external context information. ### Claim 4 (depends on 2) A method according to claim 2 including the further step of: selectively manipulating one or more of the audio streams to modify spatial properties of associated audio objects of the one or more audio streams. ### Claim 5 (depends on 4) A method according to claim 4 wherein the selectively manipulating is performed based at least in part on the contextual information. ### Claim 6 (depends on 4) A method according to claim 4 wherein the selectively manipulating is performed based at least in part on the external context information. ### Claim 7 (depends on 4) A method according to claim 4 in their dependence on claim 4 wherein the step of selectively manipulating one or more of the audio streams includes removing predetermined sounds based on their spatial, temporal or frequency characteristics. ### Claim 8 (depends on 4) A method according to claim 4 wherein the step of selectively manipulating one or more of the audio streams includes modifying a panning an audio object within the audio scene. ### Claim 9 (depends on 4) A method according to claim 4 wherein the step of selectively manipulating one or more of the audio streams includes modifying a perceived direction of travel of an audio object within the audio scene. ### Claim 10 (depends on 4) A method according to claim 4 wherein the step of selectively manipulating one or more of the audio streams includes modifying a background and/or foreground audio scene component. ### Claim 11 (depends on 4) A method according to claim 4 wherein the step of selectively manipulating one or more of the audio streams includes assigning to an audio object a spatial trajectory through the audio scene. ### Claim 12 (depends on 4) A method according to claim 4 wherein the step of selectively manipulating one or more of the audio streams includes modifying a perceived velocity of an audio object through the audio scene. ### Claim 13 (depends on 2) A method according to claim 2 wherein the external context information includes additional audio or video data relevant to the audio scene. ### Claim 14 (depends on 2) A method according to claim 2 wherein the external context information includes control input from a user. ### Claim 15 (depends on 2) A method according to claim 2 wherein the external context information includes one or more context mode settings relating to a theme of the audio scene. ### Claim 16 (depends on 1) A method according to claim 1 wherein the contextual information includes an object type. ### Claim 17 (depends on 1) A method according to claim 1 wherein the spatial properties includes one or more of size, shape, position, coherence, direction of travel, velocity or acceleration of an audio object relative to the spatial configuration. ### Claim 18 (depends on 1) A method according to claim 1 wherein the audio objects include one or more of voice, ambient sounds, instruments and noise. ### Claim 19 (depends on 1) A method according to claim 1 wherein the audio input includes a plurality of audio channels, and wherein the step of defining respective audio streams includes performing a beamforming technique on the plurality of audio channels. ### Claim 20 (depends on 1) A method according to claim 1 wherein the step of defining respective audio streams includes suppressing specific audio components. ### Claim 21 (depends on 1) A method according to claim 1 wherein the audio input includes a plurality of audio channels, and wherein performing spatial audio analysis includes performing one or more of beamforming, audio event detection, level estimation, spatial clustering, spatial classification and temporal data analysis. ### Claim 22 (depends on 1) A method according to claim 1 wherein the step of performing effects processing is automated without user input. ### Claim 23 (depends on 1) A method according to claim 1 wherein the effects processing further includes, for a given audio stream, performing one or more of equalization, tremolo, vibrato, chorus, distortion, harmonization, vocoder analysis, autotuning, delaying, applying or adjusting echo and applying or adjusting reverb. ### Claim 24 (depends on 1) The method according to claim 1 wherein the modified object-based audio signal has a different number of audio streams than the object-based audio signal. ### Claim 25 (depends on 1) A method according to claim 1 wherein the one or more audio channels are directly input from an array of microphones. ### Claim 26 (depends on 1) A method according to claim 1 wherein the encoded signal is encoded using an encoding method determined based on a type of audio objects detected in the audio input. ### Claim 27 (depends on 1) A computer-based system including a processor configured to perform the method according to claim 1. ### Claim 28 (depends on 27) The computer-based system according to claim 27 including a user interface to facilitate selection of particular audio streams. ### Claim 29 (depends on 28) The computer-based system according to claim 28 wherein the user interface is further adapted to facilitate application of particular audio effects. ### Claim 30 A system for creating an object-based audio signal from an audio input, the audio input including one or more audio channels that are recorded to collectively define an audio scene, the one or more audio channels having been captured from a respective one or more spatially separated microphones disposed in a stable spatial configuration, the system including: an input port for receiving the audio input; and a processor configured to: perform spatial analysis on the one or more audio channels to identify one or more audio objects within the audio scene; determine contextual information relating to the one or more audio objects, wherein the contextual information includes spatial properties of the one or more audio objects; and define respective audio streams including audio data relating to at least one of the identified one or more audio objects; wherein the processor is further configured to: perform effects processing on the one or more audio streams to generate a modified object-based audio signal, wherein the effects processing is based on the contextual information and on external context information that is external to the audio signal, and wherein the effects processing includes performing Doppler frequency shifting for a given audio stream based on the spatial properties included in the contextual information; and wherein the system further comprises a mastering module for mastering and outputting the modified object-based audio signal as an encoded signal, the modified object-based audio signal including the audio streams and the contextual information. ### Claim 31 (depends on 30) A system according to claim 30 including a user interface to facilitate selection of particular audio streams.