# US 12,003,953 B2 · Physics-based audio and haptic synthesis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,953 B2 - Document ID: US12003953B2 - Status: Granted - Kind: B2 - Application number: 18/181,920 - Filed: 2023-03-10 - Published: 2023-07-06 - Granted: 2024-06-04 - Earliest claimed: 2019-12-20 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Colby Nelson Leider; Justin Dan Mathew; Michael Z. Land; Blaine Ivin Wood; Jung-Suk Lee; Anastasia Andreyevna Tajik; Jean-Marc Jot - Web page: https://patentgrep.com/patent/US12003953B2 ## CPC - H04S 7/303: {Tracking of listener position or orientation} - A63F 13/285: Generating tactile feedback signals via the game input device, e.g. force feedback - A63F 13/577: using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A63F 2300/8082: Virtual reality - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S: STEREOPHONIC SYSTEMS - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Disclosed herein are systems and methods for generating and presenting virtual audio for mixed reality systems. A method may include determining a collision between a first object and a second object, wherein the first object comprises a first virtual object. A memory storing one or more audio models can be accessed. It can be determined if the one or more audio models stored in the memory comprises an audio model corresponding to the first object. In accordance with a determination that the one or more audio models comprises an audio model corresponding to the first object, an audio signal can be synthesized, wherein the audio signal is based on the collision and the audio model corresponding to the first object, and the audio signal can be presented to a user via a speaker of a head-wearable device. In accordance with a determination that the one or more audio models does not comprise an audio model corresponding to the first object, an acoustic property of the first object can be determined, a custom audio model based on the acoustic property of the first object can be generated, an audio signal can be synthesized, wherein the audio signal is based on the collision and the custom audio model, and the audio signal can be presented, via a speaker of a head-wearable device, to a user. ## Claims ### Claim 1 A method comprising: presenting, via a transmissive display of a wearable head device, a view of a mixed reality environment; determining a collision between a first object and a second object in the mixed reality environment, wherein the first object comprises at least one of a first virtual object and a first real object; accessing a memory storing acoustic data; determining whether the acoustic data stored in the memory comprises first acoustic data corresponding to the first object; in accordance with a determination that the acoustic data stored in the memory comprises the first acoustic data corresponding to the first object: determining a first audio signal based on the collision and based further on the first acoustic data corresponding to the first object, and concurrently with presenting the view of the mixed reality environment via the transmissive display, presenting the first audio signal via a speaker of the wearable head device; and in accordance with a determination that the acoustic data stored in the memory does not comprise the first acoustic data corresponding to the first object: determining an acoustic property of the first object, determining a second audio signal based on the collision and based further on the acoustic property, and concurrently with presenting the view of the mixed reality environment via the transmissive display, presenting the second audio signal via the speaker of the wearable head device. ### Claim 2 (depends on 1) The method of claim 1, wherein the second object comprises a second virtual object. ### Claim 3 (depends on 1) The method of claim 1, wherein the second object comprises a second real object. ### Claim 4 (depends on 3) The method of claim 3, further comprising: receiving, via one or more sensors of the wearable head device, sensor data associated with the second real object; determining an acoustic characteristic of the second object based on the sensor data; and generating an audio model corresponding to the second object based on the acoustic characteristic. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the first audio signal comprises determining a material of the first object and applying a property of the material as input to an audio synthesizer. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the first audio signal comprises: receiving a first property of the first object from a database; observing a second property of the first object; and applying the first property and the second property as input to an audio synthesizer. ### Claim 7 A system comprising: a wearable head device comprising: a transmissive display; a speaker; and one or more sensors; and one or more processors configured to perform a method comprising: presenting, via the transmissive display, a view of a mixed reality environment; determining a collision between a first object and a second object in the mixed reality environment, wherein the first object comprises at least one of a first virtual object and a first real object; accessing a memory storing acoustic data; determining whether the acoustic data stored in the memory comprises first acoustic data corresponding to the first object; in accordance with a determination that the acoustic data stored in the memory comprises the first acoustic data corresponding to the first object: determining a first audio signal based on the collision and based further on the first acoustic data corresponding to the first object, and concurrently with presenting the view of the mixed reality environment via the transmissive display, presenting the first audio signal via the speaker; and in accordance with a determination that the acoustic data stored in the memory does not comprise the first acoustic data corresponding to the first object: determining an acoustic property of the first object, determining a second audio signal based on the collision and based further on the acoustic property, and concurrently with presenting the view of the mixed reality environment via the transmissive display, presenting the second audio signal via the speaker. ### Claim 8 (depends on 7) The system of claim 7, wherein the second object comprises a second virtual object. ### Claim 9 (depends on 7) The system of claim 7, wherein the second object comprises a second real object. ### Claim 10 (depends on 9) The system of claim 9, wherein the method further comprises: receiving, via the one or more sensors, sensor data associated with the second real object; determining an acoustic characteristic of the second object based on the sensor data; and generating an audio model corresponding to the second object based on the acoustic characteristic. ### Claim 11 (depends on 7) The system of claim 7, wherein determining the first audio signal comprises determining a material of the first object and applying a property of the material as input to an audio synthesizer. ### Claim 12 (depends on 7) The system of claim 7, wherein determining the first audio signal comprises: receiving a first property of the first object from a database; observing a second property of the first object; and applying the first property and the second property as input to an audio synthesizer. ### Claim 13 A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising: presenting, via a transmissive display of a wearable head device, a view of a mixed reality environment; determining a collision between a first object and a second object in the mixed reality environment, wherein the first object comprises one or more of a first virtual object and a first real object; accessing a memory storing acoustic data; determining whether the acoustic data stored in the memory comprises first acoustic data corresponding to the first object; in accordance with a determination that the acoustic data stored in the memory comprises the first acoustic data corresponding to the first object: determining a first audio signal based on the collision and based further on the first acoustic data corresponding to the first object, and concurrently with presenting the view of the mixed reality environment via the transmissive display, presenting the first audio signal via a speaker of the wearable head device; and in accordance with a determination that the acoustic data stored in the memory does not comprise the first acoustic data corresponding to the first object: determining an acoustic property of the first object, determining a second audio signal based on the collision and based further on the acoustic property, and concurrently with presenting the view of the mixed reality environment via the transmissive display, presenting the second audio signal via the speaker of the wearable head device. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the second object comprises a second virtual object. ### Claim 15 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the second object comprises a second real object. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: receiving, via one or more sensors of the wearable head device, sensor data associated with the second real object; determining an acoustic characteristic of the second object based on the sensor data; and generating an audio model corresponding to the second object based on the acoustic characteristic. ### Claim 17 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein determining the first audio signal comprises: receiving a first property of the first object from a database; observing a second property of the first object; and applying the first property and the second property as input to an audio synthesizer.