# US 2020/0404444 A1 · Audio processing to modify a spatial extent of a sound object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0404444 A1 - Document ID: US20200404444A1 - Status: Published application - Kind: A1 - Application number: 17/014,245 - Filed: 2020-09-08 - Published: 2020-12-24 - Earliest claimed: 2016-12-01 - Assignees: Nokia Technologies Oy - Applicants: Nokia Technologies Oy - Inventors: Antti Eronen; Jussi Leppänen; Francesco Cricri; Arto Lehtiniemi - Web page: https://patentgrep.com/patent/US20200404444A1 ## CPC - H04S 7/40: {Visual indication of stereophonic sound image} - G06F 18/24: Classification techniques - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)} - H04S 7/30: {Control circuits for electronic adaptation of the sound field} - H04S 7/303: {Tracking of listener position or orientation} - G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing - H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S: STEREOPHONIC SYSTEMS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method comprising: causing analysis of a portion of a visual scene; causing modification of a first sound object to modify a spatial extent of the first sound object in dependence upon the analysis of the portion of the visual scene corresponding to the first sound object; and causing rendering of the visual scene and the corresponding sound scene including of the modified first sound object with modified spatial extent. ## Claims ### Claim 1 A method comprising: receiving a visual scene comprising at least one visual object and receiving at least one corresponding sound object; determining at least one characteristic associated with the at least one visual object; processing the at least one sound object to modify a spatial extent of the at least one sound object in dependence upon the determined at least one characteristic; and generating a modified scene comprising the at least one visual object and the spatially extended at least one sound object for rendering. ### Claim 2 (depends on 1) A method as claimed in claim 1, wherein determining at least one characteristic comprises determining at least one of a size or location of the at least one visual object. ### Claim 3 (depends on 1) A method as claimed in claim 1, wherein determining at least one characteristic comprises determining a classification of the at least one visual object. ### Claim 4 (depends on 1) A method as claimed in claim 1, wherein processing the at least one sound object comprises causing splitting the at least one sound object that corresponds to the at least one visual object into sound sub-objects; and controlling positioning and spatial extent of the sound sub-objects. ### Claim 5 (depends on 4) A method as claimed in claim 4, further comprising causing classification of the at least one visual object into visual sub-objects, wherein controlling the positioning and spatial extent of the sound sub-objects so as to correspond with the visual sub-objects. ### Claim 6 (depends on 1) A method as claimed in claim 1, wherein the spatial extent of the at least one sound object, following modification, is the same as a spatial extent of the at least one visual object that corresponds to the at least one sound object. ### Claim 7 (depends on 1) A method as claimed in claim 1, wherein the spatial extent of the at least one sound object is one or more lengths along one or more vectors in a sound space; and wherein a spatial extent of the at least one visual object that corresponds to the at least one sound object is one or more lengths along one or more vectors in a visual space. ### Claim 8 (depends on 1) A method as claimed in claim 1, wherein processing the at least one sound object comprises adapting the at least one sound object so that the at least one sound object is no longer rendered as a point sound source. ### Claim 9 (depends on 1) A method as claimed in claim 1, wherein processing the at least one sound object comprises dividing an audio signal representing the at least one sound object into different frequency bands to create multiple spectrally-limited audio signals and placing each of the multiple spectrally-limited audio signals into different spatial audio channels. ### Claim 10 (depends on 9) A method as claimed in claim 9, wherein the multiple spectrally-limited audio signals are placed randomly into a set of spatial audio channels or wherein the multiple spectrally-limited audio signals are placed in spatial audio channels according to predefined rules. ### Claim 11 An apparatus comprises at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to: receive a visual scene comprising at least one visual object and receiving at least one corresponding sound object; determine at least one characteristic associated with the at least one visual object; processing the at least one sound object to modify a spatial extent of the at least one sound object in dependence upon the determined at least one characteristic; and generate a modified scene comprising the at least one visual object and the spatially extended at least one sound object for rendering. ### Claim 12 (depends on 11) An apparatus as claimed in claim 11, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to determine at least one characteristic by determining at least one of a size or location of the at least one visual object. ### Claim 13 (depends on 11) An apparatus as claimed in claim 11, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to determine at least one characteristic by determining a classification of the at least one visual object. ### Claim 14 (depends on 11) An apparatus as claimed in claim 11, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to process the at least one sound object by causing splitting the at least one sound object that corresponds to the at least one visual object into sound sub-objects; and controlling positioning and spatial extent of the sound sub-objects. ### Claim 15 (depends on 14) An apparatus as claimed in claim 14, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to cause classification of the at least one visual object into visual sub-objects, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to control the positioning and spatial extent of the sound sub-objects so as to correspond with the visual sub-objects. ### Claim 16 (depends on 11) An apparatus as claimed in claim 11, wherein the spatial extent of the at least one sound object, following modification, is the same as a spatial extent of the at least one visual object that corresponds to the at least one sound object. ### Claim 17 (depends on 11) An apparatus as claimed in claim 11, wherein the spatial extent of the at least one sound object is one or more lengths along one or more vectors in a sound space; and wherein a spatial extent of the at least one visual object that corresponds to the at least one sound object is one or more lengths along one or more vectors in a visual space. ### Claim 18 (depends on 11) An apparatus as claimed in claim 11, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to process the at least one sound object by adapting the at least one sound object so that the at least one sound object is no longer rendered as a point sound source. ### Claim 19 (depends on 11) An apparatus as claimed in claim 11, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to process the at least one sound object by dividing an audio signal representing the at least one sound object into different frequency bands to create multiple spectrally-limited audio signals and placing each of the multiple spectrally-limited audio signals into different spatial audio channels. ### Claim 20 (depends on 19) An apparatus as claimed in claim 19, wherein the multiple spectrally-limited audio signals are placed randomly into a set of spatial audio channels or wherein the multiple spectrally-limited audio signals are placed in spatial audio channels according to predefined rules.