# US 11,900,912 B2 · Surface appropriate collisions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,900,912 B2
- Document ID: US11900912B2
- Status: Granted
- Kind: B2
- Application number: 18/185,272
- Filed: 2023-03-16
- Published: 2023-07-13
- Granted: 2024-02-13
- Earliest claimed: 2020-05-29
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Anastasia Andreyevna Tajik; Colby Nelson Leider; Omer Perry
- Web page: https://patentgrep.com/patent/US11900912B2

## CPC

- G10K 15/02: Synthesis of acoustic waves (synthesis of speech G10L13/00)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 20/20: in augmented reality scenes
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 40/193: {Preprocessing; Feature extraction}
- G10L 25/57: for processing of video signals
- H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor
- H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04S 7/303: {Tracking of listener position or orientation}
- G06V 20/44: {Event detection}
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- G10K: SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

Disclosed herein are systems and methods for presenting an audio signal associated with presentation of a virtual object colliding with a surface. The virtual object and the surface may be associated with a mixed reality environment. Generation of the audio signal may be based on at least one of an audio stream from a microphone and a video stream form a sensor. In some embodiments, the collision between the virtual object and the surface is associated with a footstep on the surface.

## Claims

### Claim 1

A system comprising:
a microphone;
a display;
an audio output; and
one or more processors configured to execute a method comprising:
at a first time, detecting a footstep, of a user, on a surface;
in accordance with said detecting the footstep, determining, via a detection of the footstep by the microphone, an acoustic property of the surface;
at a second time, detecting a collision of a virtual object with the surface;
in accordance with said detecting the collision:
presenting to the user, on the display, a view of the collision;
generating an audio signal based on the collision and based further on the acoustic property of the surface; and
presenting to the user, via the audio output, concurrently with presenting the view of the collision, the audio signal.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a sensor, wherein said detecting the footstep is based on movement data of the user captured by the sensor.

### Claim 3 (depends on 1)

The system of claim 1, further comprising an inertial measurement unit, wherein said detecting the footstep is based on inertial data of the user captured by the inertial measurement unit.

### Claim 4 (depends on 1)

The system of claim 1, further comprising a wearable auxiliary device and a wearable head device, wherein:
the wearable auxiliary device comprises the one or more processors, and
the auxiliary device is coupled to the wearable head device.

### Claim 5 (depends on 1)

The system of claim 1, wherein the system further comprises a wearable head device and wherein the method further comprises:
in accordance with said detecting the footstep:
determining a position of the wearable head device of the system; and
determining a position of the footstep based on the position of the wearable head device of the system;
wherein said generating the audio signal is further based on the position of the footstep.

### Claim 6 (depends on 5)

The system of claim 5, wherein:
the view of the collision comprises a view of the virtual object colliding with the surface at the position of the footstep.

### Claim 7 (depends on 1)

The system of claim 1, wherein said generating the audio signal is further based on at least one of a physical model of the surface and a physical model of the virtual object.

### Claim 8 (depends on 1)

The system of claim 1, wherein said generating the audio signal is further based on an analysis of an audio stream associated with the audio signal.

### Claim 9 (depends on 1)

The system of claim 1, wherein the method further comprises associating the audio signal with the surface.

### Claim 10 (depends on 1)

The system of claim 1, wherein the audio signal corresponds to the collision of the virtual object with the surface.

### Claim 11 (depends on 1)

The system of claim 1, wherein:
said generating the audio signal is further based on a stored audio signal.

### Claim 12 (depends on 1)

The system of claim 1, wherein the method further comprises:
detecting a second collision with a second surface;
in accordance with the detection of the second collision:
identifying a portion of an audio stream associated with the audio signal;
generating a second audio signal based on the portion of the audio stream and based further on the acoustic property of the surface; and
presenting to the user, via the audio output, the second audio signal.

### Claim 13 (depends on 1)

The system of claim 1, wherein:
the audio signal is associated with a video stream,
the video stream includes information associated with the surface, and
said generating the audio signal is based further on the information.

### Claim 14 (depends on 1)

The system of claim 1, wherein the virtual object comprises a foot of a virtual character.

### Claim 15 (depends on 1)

The system of claim 1, wherein:
the acoustic property comprises a property of a material of the surface.

### Claim 16 (depends on 15)

The system of claim 15, wherein said determining the acoustic property is based on a coefficient of absorption of the material.

### Claim 17

A method comprising:
at a first time, detecting a footstep, of a user, on a surface;
in accordance with said detecting the footstep, determining, via a detection of the footstep by a microphone, an acoustic property of the surface;
at a second time, detecting a collision of a virtual object with the surface;
in accordance with said detecting the collision:
presenting to the user, on a display, a view of the collision;
generating an audio signal based on the collision and based further on the acoustic property of the surface; and
presenting to the user, via an audio output, concurrently with presenting the view of the collision, the audio signal.

### Claim 18 (depends on 17)

The method of claim 17, wherein said detecting the footstep is based on movement data of the user captured by a sensor.

### Claim 19 (depends on 17)

The method of claim 17, wherein said detecting the footstep is based on inertial data of the user captured by an inertial measurement unit.

### Claim 20

A non-transitory computer-readable medium storing one or more instructions, which, when executed by one or more processors, cause the one or more processors to perform a method comprising:
at a first time, detecting a footstep, of a user, on a surface;
in accordance with said detecting the footstep, determining, via a detection of the footstep by a microphone, an acoustic property of the surface;
at a second time, detecting a collision of a virtual object with the surface;
in accordance with said detecting the collision:
presenting to the user, on a display, a view of the collision;
generating an audio signal based on the collision and based further on the acoustic property of the surface; and
presenting to the user, via an audio output, concurrently with presenting the view of the collision, the audio signal.