# US 11,443,487 B2 · Methods, apparatus, systems, computer programs for enabling consumption of virtual content for mediated reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,487 B2 - Document ID: US11443487B2 - Status: Granted - Kind: B2 - Application number: 16/625,900 - Filed: 2018-06-19 - Published: 2020-05-21 - Granted: 2022-09-13 - Earliest claimed: 2017-06-30 - Assignees: Nokia Technologies Oy - Applicants: Nokia Technologies Oy - Inventors: Jussi Leppänen; Arto Lehtiniemi; Sujeet Shyamsundar Mate; Antti Eronen - Web page: https://patentgrep.com/patent/US11443487B2 ## CPC - G06T 19/003: {Navigation within 3D models or images} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/16: Sound input; Sound output (speech processing G10L) - H04S 7/303: {Tracking of listener position or orientation} - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04S: STEREOPHONIC SYSTEMS ## Abstract A method comprising: switching between a first mode for controlling a relative position of a virtual sound object from a notional listener in a virtual space in dependence upon a point of view of the notional listener and a second mode for controlling the relative position of the virtual sound object from the notional listener in the virtual space in dependence upon the point of view of the notional listener, wherein during the first mode there is a higher dependency of the relative position of the virtual sound object from the notional listener upon a currently determined relative position of the virtual sound object from the notional listener in the virtual space based on the current measured location of a user in a real space and during the second mode there is a lower dependency of the relative position of the virtual sound object from the notional listener upon a currently determined relative position of the virtual sound object from the notional listener in the virtual space based on the current measured location of the user in the real space. ## Claims ### Claim 1 An apparatus comprising at least one processor; and at least one non-transitory memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: switch between a first mode for controlling a relative position of a virtual sound object from a notional listener in a virtual space in dependence upon a point of view of the notional listener, and a second mode for controlling the relative position of the virtual sound object from the notional listener in the virtual space in dependence upon the point of view of the notional listener; during the first mode, determine the controlled relative position of the virtual sound object from the notional listener with a first dependency upon a currently determined relative position of the virtual sound object from the notional listener in the virtual space that is based on a current measured location of a user in a real space; during the second mode, determine the controlled relative position of the virtual sound object from the notional listener with a second dependency upon the currently determined relative position of the virtual sound object from the notional listener in the virtual space that is based on the current measured location of the user in the real space, wherein a transition from the first mode to the second mode is configured to occur when a level of uncertainty associated with measurement of the current measured location of the user in the real space is above a threshold; and during the second mode, damp changes in the controlled relative position of the virtual sound object from the notional listener, in response to changes in the current measured location of the user in the real space, in comparison to changes in the controlled relative position of the virtual sound object during the first mode in response to the changes in the current measured location of the user in the real space, but still dependent upon changes in orientation of the notional listener in the virtual space, wherein the first dependency is a higher dependency than the second dependency. ### Claim 2 (depends on 1) An apparatus as claimed in claim 1, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform: during the first mode, determine the controlled relative position of the virtual sound object from the notional listener with a third dependency upon a currently determined orientation of the notional listener in the virtual space that is based on a current measured point of view of the user in the real space; and during the second mode, determine the controlled relative position of the virtual sound object from the notional listener with a fourth dependency upon the currently determined orientation of the notional listener in the virtual space that is based on the current measured point of view of the user in the real space, wherein the third dependency is a higher dependency than the fourth dependency. ### Claim 3 (depends on 1) An apparatus as claimed in claim 1, wherein during the second mode: the controlled relative position of the virtual sound object from the notional listener is not configured to be switched between a left hand side and a right hand side of the notional listener in response to changes in the current measured location of the user in the real space, and the controlled relative position of the virtual sound object from the notional listener is configured to be switched between the left hand side and the right hand side of the notional listener, in response to changes in the orientation of the notional listener in the virtual space. ### Claim 4 (depends on 1) An apparatus as claimed in claim 1, wherein during the second mode the controlled relative position of the virtual sound object from the notional listener is configured to be independent of the current measured location of user in the real space but still dependent upon the changes in the orientation of the notional listener in the virtual space. ### Claim 5 (depends on 1) An apparatus as claimed in claim 1, wherein during the second mode the controlled relative position of the virtual sound object from the notional listener is configured to be locked to the orientation of the notional listener in the virtual space. ### Claim 6 (depends on 1) An apparatus as claimed in claim 1, wherein the transition from the first mode to the second mode is further configured to occur when a distance of the virtual sound object from the notional listener in the virtual space decreases below a threshold, wherein, during the second mode, the determined controlled relative position of the virtual sound object from the notional listener is configured to maintain a position of the virtual sound object relative to the notional listener at a time of the transition. ### Claim 7 (depends on 6) An apparatus as claimed in claim 6, wherein a transition from the second mode to the first mode is configured to occur when the distance of the virtual sound object from the notional listener in the virtual space increases above a further threshold. ### Claim 8 (depends on 1) An apparatus as claimed in claim 1, wherein the transition from the first mode to the second mode is further configured to occur when a variation of the currently determined relative position of the virtual sound object from the notional listener over time satisfies a criterion. ### Claim 9 (depends on 1) An apparatus as claimed in claim 1, wherein on switching from the second mode to the first mode, the change in the controlled relative position of the virtual sound object from the notional listener is configured to be smoothed over time. ### Claim 10 (depends on 1) An apparatus as claimed in claim 1, wherein the controlled relative position of the virtual sound object from the notional listener is configured to be dependent upon the currently determined relative position of the virtual sound object from the notional listener in the virtual space based on the current measured location of the user in the real space, and a model-based location of the user. ### Claim 11 (depends on 10) An apparatus as claimed in claim 10, wherein the model-based location of the user is configured to be produced using a Kalman filter. ### Claim 12 (depends on 11) An apparatus as claimed in claim 11, wherein the model-based location of the user is configured to be produced using the Kalman filter operating at a rate less than 20 Hz. ### Claim 13 A method comprising: switching between a first mode for controlling a relative position of a virtual sound object from a notional listener in a virtual space in dependence upon a point of view of the notional listener, and a second mode for controlling the relative position of the virtual sound object from the notional listener in the virtual space in dependence upon the point of view of the notional listener; during the first mode, determining the controlled relative position of the virtual sound object from the notional listener with a first dependency upon a currently determined relative position of the virtual sound object from the notional listener in the virtual space that is based on a current measured location of a user in a real space; during the second mode, determining the controlled relative position of the virtual sound object from the notional listener with a second dependency upon the currently determined relative position of the virtual sound object from the notional listener in the virtual space that is based on the current measured location of the user in the real space, wherein a transition from the first mode to the second mode occurs when a level of uncertainty associated with measurement of the current measured location of the user in the real space is above a threshold; and during the second mode, damping changes in the controlled relative position of the virtual sound object from the notional listener, in response to changes in the current measured location of the user in the real space, in comparison to changes in the controlled relative position of the virtual sound object during the first mode in response to the changes in the current measured location of the user in the real space, but still dependent upon changes in orientation of the notional listener in the virtual space, wherein the first dependency is a higher dependency than the second dependency. ### Claim 14 (depends on 13) A method as claimed in claim 13, further comprising: during the first mode, determining the controlled relative position of the virtual sound object from the notional listener with a third dependency upon a currently determined orientation of the notional listener in the virtual space that is based on a current measured point of view of the user in the real space; and during the second mode, determining the controlled relative position of the virtual sound object from the notional listener with a fourth dependency upon the currently determined orientation of the notional listener in the virtual space that is based on the current measured point of view of the user in the real space, wherein the third dependency is a higher dependency than the fourth dependency. ### Claim 15 (depends on 13) A method as claimed in claim 13, wherein during the second mode the controlled relative position of the virtual sound object from the notional listener cannot switch between a left hand side and a right hand side of the notional listener except as a consequence of the changes in the orientation of the notional listener in the virtual space. ### Claim 16 (depends on 13) A method as claimed in claim 13, wherein during the second mode the controlled relative position of the virtual sound object from the notional listener is independent of the current measured location of user in the real space but still dependent upon changes in the orientation of the notional listener in the virtual space. ### Claim 17 (depends on 13) A method as claimed in claim 13, wherein during the second mode the controlled relative position of the virtual sound object from the notional listener is locked to the orientation of the notional listener in the virtual space. ### Claim 18 (depends on 13) A method as claimed in claim 13, wherein the transition from the first mode to the second mode occurs when a distance of the virtual sound object from the notional listener in the virtual space decreases below a threshold. ### Claim 19 (depends on 13) A method as claimed in claim 13, wherein the transition from the first mode to the second mode occurs when a variation of the currently determined relative position of the virtual sound object from the notional listener over time satisfies a criterion. ### Claim 20 A non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following: switch between a first mode for controlling a relative position of a virtual sound object from a notional listener in a virtual space in dependence upon a point of view of the notional listener, and a second mode for controlling the relative position of the virtual sound object from the notional listener in the virtual space in dependence upon the point of view of the notional listener; during the first mode enable determining of the controlled relative position of the virtual sound object from the notional listener with a first dependency upon a currently determined relative position of the virtual sound object from the notional listener in the virtual space that is based on a current measured location of a user in a real space; during the second mode enable determining of the controlled relative position of the virtual sound object from the notional listener with a second dependency upon the currently determined relative position of the virtual sound object from the notional listener in the virtual space that is based on the current measured location of the user in the real space, wherein a transition from the first mode to the second mode is configured to occur when a level of uncertainty associated with measurement of the current measured location of the user in the real space is above a threshold; and during the second mode, enable damping of changes in the controlled relative position of the virtual sound object from the notional listener, in response to changes in the current measured location of the user in the real space, in comparison to changes in the controlled relative position of the virtual sound object during the first mode in response to the changes in the current measured location of the user in the real space, but still dependent upon changes in orientation of the notional listener in the virtual space, wherein the first dependency is a higher dependency than the second dependency.