# US 2021/0082191 A1 · Mixed reality musical instrument

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082191 A1
- Document ID: US20210082191A1
- Status: Published application
- Kind: A1
- Application number: 16/970,323
- Filed: 2019-02-15
- Published: 2021-03-18
- Earliest claimed: 2018-02-15
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Anastasia Andreyevna TAJIK
- Web page: https://patentgrep.com/patent/US20210082191A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G10H 1/0008: {Associated control or indicating means}
- G10H 1/0066: {using a MIDI interface}
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4524: {involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00)}
- H04N 21/8106: {involving special audio data, e.g. different tracks for different languages}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04R 1/1091: {Details not provided for in groups H04R1/1008 - H04R1/1083}
- H04S 7/303: {Tracking of listener position or orientation}
- H04S 7/304: {For headphones}
- G10H 2210/325: Musical pitch modification
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A method is disclosed, the method comprising the steps of identifying a first real object in a mixed reality environment, the mixed reality environment having a user; identifying a second real object in the mixed reality environment; generating, in the mixed reality environment, a first virtual object corresponding to the second real object; identifying, in the mixed reality environment, a collision between the first real object and the first virtual object; determining a first attribute associated with the collision; determining, based on the first attribute, a first audio signal corresponding to the collision; and presenting to the user, via one or more speakers, the first audio signal.

## Claims

### Claim 1

A method comprising:
identifying a first real object in a mixed reality environment, the mixed reality environment having a user;
identifying a second real object in the mixed reality environment;
generating, in the mixed reality environment, a first virtual object corresponding to the second real object;
identifying, in the mixed reality environment, a collision between the first real object and the first virtual object;
determining a first attribute associated with the collision;
determining, based on the first attribute, a first audio signal corresponding to the collision; and
presenting to the user, via a first speaker, the first audio signal.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the collision comprises predicting a collision based on a position of the first real object and a velocity of the first real object.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first attribute is a pitch associated with the virtual object, and determining a first audio signal comprises generating an audio signal corresponding to the pitch.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first attribute comprises a material associated with the virtual object, and determining the first audio signal comprises:
generating an audio signal, and
modifying the audio signal based on an acoustic property of the material.

### Claim 5 (depends on 1)

The method of claim 1, further comprising presenting, concurrently on a display, the virtual object and a view of the second real object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first real object comprises a sensor, and identifying the collision comprises predicting a collision based on an output of the sensor.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the first audio signal comprises:
generating a MIDI parameter based on the first attribute; and
generating, using a sound engine, the first audio signal based on the MIDI parameter.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
identifying a third real object in the mixed reality environment;
generating, in the mixed reality environment, a second virtual object corresponding to the third real object;
identifying, in the mixed reality environment, a collision between the first virtual object and the second virtual object;
determining a second attribute associated with the collision between the first virtual object and the second virtual object;
determining, based on the second attribute, a second audio signal corresponding to the collision between the first virtual object and the second virtual object; and
presenting to the user, via the first speaker, the second audio signal.

### Claim 9

A system comprising:
a wearable head device including:
a display for displaying a virtual environment, the display comprising a transmissive eyepiece through which a real environment is visible; and
a speaker; and
one or more processors configured to perform:
identifying a first real object in the real environment;
identifying a second real object in the real environment;
generating, in the virtual environment, a first virtual object corresponding to the second real object;
identifying, in the virtual environment, a collision between the first real object and the first virtual object;
determining a first attribute associated with the collision;
determining, based on the first attribute, a first audio signal corresponding to the collision; and
presenting, to a user of the wearable head device, via the speaker, the first audio signal.

### Claim 10 (depends on 9)

The system of claim 9, wherein identifying the collision comprises predicting a collision based on a position of the first real object and a velocity of the first real object.

### Claim 11 (depends on 9)

The system of claim 9, wherein the first attribute is a pitch associated with the virtual object, and determining a first audio signal comprises generating an audio signal corresponding to the pitch.

### Claim 12 (depends on 9)

The system of claim 9, wherein the first attribute comprises a material associated with the virtual object, and determining the first audio signal comprises:
generating an audio signal, and
modifying the audio signal based on an acoustic property of the material.

### Claim 13 (depends on 9)

The system of claim 9, the one or more processors further configured to perform presenting, concurrently on the display, the virtual object and a view of the second real object.

### Claim 14 (depends on 9)

The system of claim 9, wherein the first real object comprises a sensor, and identifying the collision comprises predicting a collision based on an output of the sensor.

### Claim 15 (depends on 9)

The system of claim 9, wherein determining the first audio signal comprises:
generating a MIDI parameter based on the first attribute; and
generating, using a sound engine, the first audio signal based on the MIDI parameter.

### Claim 16 (depends on 9)

The system of claim 9, wherein the wearable head device further includes a sensor, and the first attribute is determined based on an output of the sensor.

### Claim 17 (depends on 9)

The system of claim 9, the one or more processors further configured to perform:
identifying a third real object in the real environment;
generating, in the virtual environment, a second virtual object corresponding to the third real object;
identifying, in the virtual environment, a collision between the first virtual object and the second virtual object;
determining a second attribute associated with the collision between the first virtual object and the second virtual object;
determining, based on the second attribute, a second audio signal corresponding to the collision between the first virtual object and the second virtual object; and
presenting, to the user, via the speaker, the second audio signal.

### Claim 18 (depends on 8)

The method of claim 8, wherein the first real object comprises a sensor, and identifying the collision between the first virtual object and the second virtual object comprises predicting a virtual collision based on an output of the sensor.

### Claim 19 (depends on 17)

The system of claim 17, wherein the first real object comprises a sensor, and identifying the collision between the first virtual object and the second virtual object comprises predicting a virtual collision based on an output of the sensor.

### Claim 20

A non-transitory computer-readable medium containing instructions which when executed by one or more processors cause the one or more processors to perform a method of presenting an audio signal to a user of a wearable head device, the method comprising:
identifying a first real object in a mixed reality environment, the mixed reality environment having a user;
identifying a second real object in the mixed reality environment;
generating, in the mixed reality environment, a first virtual object corresponding to the second real object;
identifying, in the mixed reality environment, a collision between the first real object and the first virtual object;
determining a first attribute associated with the collision;
determining, based on the first attribute, a first audio signal corresponding to the collision; and
presenting to the user, via a first speaker, the first audio signal.
