# US 2018/0324539 A1 · Method and system of improving detection of environmental sounds in an immersive environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324539 A1
- Document ID: US20180324539A1
- Status: Published application
- Kind: A1
- Application number: 15/589,986
- Filed: 2017-05-08
- Published: 2018-11-08
- Earliest claimed: 2017-05-08
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Andrew William LOVITT
- Web page: https://patentgrep.com/patent/US20180324539A1

## CPC

- G02B 27/017: {Head mounted}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04S: STEREOPHONIC SYSTEMS
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

An improved method and system for enhancing a user's experience within an immersive environment is disclosed. Solutions and implementations provided improve the user's ability to both hear and identify sounds from outside the user's immersive experience. Implementations include receiving audio and sensor input from the environment, analyzing the received input and adjusting either the display or audio outputs generated for the user based in part on pre-determined policies.

## Claims

### Claim 1

A method of improving a user's experience in an immersive environment comprising:
receiving an audio signal from an ambient environment;
recording the received audio signal;
analyzing the received audio signal to identify one or more parameters for the received audio signal;
accessing a policy database to determine if there are one or more policies in the policy database that are relevant to the audio signal being processed based at least in part on one of the identified one or more parameters;
determining a classification level for the received audio signal based at least in part on one of the one or more identified parameters of the received audio signal and at least one of the one or more policies;
determining, at least in part based on the classification level for the received audio signal, if an adjustment is needed in the immersive environment to improve the user's experience; and
upon determining the adjustment is needed, adjusting a relationship between at least one parameter of a virtual agent in the immersive environment and at least one of the identified one or more parameters.

### Claim 2 (depends on 1)

The method of claim 1, wherein at least one of the one or more parameters comprises a volume of the received audio signal.

### Claim 3 (depends on 2)

The method of claim 2, wherein adjusting the relationship includes adjusting the volume of the received audio signal with respect to a volume of the virtual agent.

### Claim 4 (depends on 2)

The method of claim 2, wherein adjusting the relationship includes adjusting the volume of the virtual agent with respect to the volume of the received audio signal.

### Claim 5 (depends on 1)

The method of claim 1, wherein at least one of the one or more parameters comprises a source for the received audio signal.

### Claim 6 (depends on 5)

The method of claim 5, wherein adjusting the relationship includes changing a location of the virtual agent with respect to a location of the source of the received audio signal.

### Claim 7 (depends on 1)

The method of claim 1, further comprising providing an output audio signal to the user based in part on the received audio signal and the adjusted relationship.

### Claim 8 (depends on 7)

The method of claim 7, wherein the output audio signal is provided to the user with a time delay with respect to the received audio signal.

### Claim 9

(canceled)

### Claim 10 (depends on 7)

The method of claim 7, wherein adjusting the relationship includes presenting the output audio signal from a location that is different from of a location of a source of the received audio signal.

### Claim 11 (depends on 10)

The method of claim 10, wherein the location from which the output audio signal is presented is further away from a location of the virtual agent than the location of the source of the received audio signal and wherein once the user turns their head towards the location from which the output audio signal is presented the output audio signal is presented from the location of the source of the received audio signal.

### Claim 12

An apparatus comprising:
a microphone for receiving an audio signal;
a display panel for displaying a virtual agent;
a speaker for transmitting an output audio signal;
a processing unit; and
a memory readable by the processing unit and comprising instructions stored thereon to cause the processing unit to:
analyze a received audio signal to identify one or more parameters for the received audio signal;
record the received audio signal;
access a policy database to determine if there are one or more policies in the policy database that are relevant to the audio signal based at least in part on one of the identified one or more parameters;
determine a classification level for the received audio signal based at least in part on one of the one or more identified parameters of the received audio signal and at least one of the one or more policies;
determine, at least in part based on the classification level, if an adjustment is needed to improve a user's experience; and
upon determining that the adjustment is needed, adjust a relationship between at least one parameter of the virtual agent and at least one of the identified one or more parameters.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the processing unit includes an audio processing unit and a sensor processing unit.

### Claim 14 (depends on 12)

The apparatus of claim 12, further comprising a sensor for receiving data about a real-world environment surrounding the apparatus.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the sensor comprises at least one of a camera, an accelerometer, and a motion sensor.

### Claim 16

A method of improving a user's experience in an immersive environment comprising:
receiving an audio signal from an ambient environment;
analyzing the received audio signal to identify one or more parameters for the received audio signal;
receiving a sensor signal from the ambient environment;
accessing a policy database to identify one or more policies in the policy database related to at least one of the one or more parameters of the received sensor signal;
determining, at least in part based on the identified one or more policies, if an adjustment is needed in the immersive environment to improve the user's experience; and
upon determining that the adjustment is needed, adjusting a relationship between at least one parameter of a virtual agent in the immersive environment and at least one of the identified one or more parameters.

### Claim 17 (depends on 16)

The method of claim 16, wherein the sensor signal comprises at least one of an image, a video, accelerometer data, motion information, or location data.

### Claim 18 (depends on 16)

The method of claim 16, further comprising analyzing the received sensor signal to identify at least one of an object or a person in the immersive environment.

### Claim 19 (depends on 16)

The method of claim 16, wherein identifying at least one of the one or more parameters includes identifying a source for the audio signal by employing a voice recognition algorithm.

### Claim 20 (depends on 16)

The method of claim 16, further comprising decomposing an auditory field of a real-world environment surrounding the immersive environment based at least in part on historical information.
