# US 2016/0260441 A1 · Real-time remodeling of user voice in an immersive visualization system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0260441 A1
- Document ID: US20160260441A1
- Status: Published application
- Kind: A1
- Application number: 14/641,174
- Filed: 2015-03-06
- Published: 2016-09-08
- Earliest claimed: 2015-03-06
- Assignees: Microsoft Technology Licensing, LLC
- Inventors: Andrew Frederick Muehlhausen; Matthew Johnston; Kasson Crooker
- Web page: https://patentgrep.com/patent/US20160260441A1

## CPC

- G10L 21/034: Automatic adjustment
- A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone
- A63F 13/54: involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06T 17/05: Geographic models
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G10L 21/003: Changing voice quality, e.g. pitch or formants
- H04S 7/304: {For headphones}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08)
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A visualization system with audio capability includes one or more display devices, one or more microphones, one or more speakers, and audio processing circuitry. While a display device displays a holographic image to a user, a microphone inputs an utterance of the user, or a sound from the user's environment, and provides it to the audio processing circuitry. The audio processing circuitry processes the utterance (or other sound) in real-time to add an audio effect associated with the image to increase realism, and outputs the processed utterance (or other sound) to the user via the speaker in real-time, with very low latency.

## Claims

### Claim 1

A method comprising:
displaying, by a head-mounted display device, a holographic image of a physical thing to a user of the head-mounted display device, over a real world view of an environment of the user;
inputting voice of the user via a microphone of the head-mounted display device, while the holographic image is being displayed to the user;
dynamically processing user voice data representing the voice of the user in real-time to incorporate an audio effect corresponding to the physical thing, while the holographic image is being displayed to the user; and
outputting, via a speaker, in real-time, sound representing the voice of the user as affected by the physical thing, based on the dynamically processed user voice data, while the holographic image is being displayed to the user.

### Claim 2 (depends on 1)

A method as recited in claim 1, wherein a latency between said inputting voice of the user and said outputting sound to the user is less than 75 milliseconds for any particular audio sample of the input voice of the user.

### Claim 3 (depends on 1)

A method as recited in claim 1, wherein the physical thing is an object or substance.

### Claim 4 (depends on 1)

A method as recited in claim 1, wherein the physical thing is a cavity or hole in an object or substance.

### Claim 5 (depends on 1)

A method as recited in claim 1, further comprising:
tracking, by the head-mounted display device, at least one of a head location or head orientation of the user;
wherein dynamically processing the user voice data is further based on at least one of the head location or head orientation of the user.

### Claim 6 (depends on 5)

A method as recited in claim 5, wherein dynamically modifying the user voice data comprises determining a directionality parameter of the modified user voice data, based on at least one of the head location or head orientation of the user.

### Claim 7 (depends on 5)

A method as recited in claim 5, further comprising:
generating, by the head-mounted display device, a surface reconstruction of the environment of the user;
wherein dynamically processing the user voice data is further based on the surface reconstruction.

### Claim 8 (depends on 1)

A method as recited in claim 1, wherein dynamically processing the user voice data comprises determining a volume parameter of the modified user voice data, based on at least one of a head location or head orientation of the user.

### Claim 9 (depends on 1)

A method as recited in claim 1, wherein dynamically processing the user voice data comprises determining a reverberation parameter of the modified user voice data, based on at least one of a head location or head orientation of the user.

### Claim 10 (depends on 1)

A method as recited in claim 1, wherein dynamically processing the user voice data comprises determining an origination direction of a simulated echo of the user's voice, based on at least one of the head location or head orientation of the user.

### Claim 11

A method of operating a head-mounted display device with audio capability, the method comprising:
displaying, by the head-mounted display device, a holographic image depicting a physical thing to a user of the head-mounted display device, such that the holographic image is superimposed over a real world view of a physical environment in which the user is located;
inputting voice of the user by using an array of microphones of the head-mounted display device, while the holographic image is being displayed to the user;
dynamically modifying user voice data representing the voice of the user to incorporate an audio effect that corresponds to an acoustic property of the physical thing, while the holographic image is being displayed to the user; and
outputting sound to the user, via a speaker of the head-mounted display device, based on the modified user voice data, to produce an audible effect applied to the voice of the user corresponding to the acoustic property of the physical thing, while the holographic image is being displayed to the user.

### Claim 12 (depends on 11)

A method as recited in claim 11, wherein the physical thing is at least one of:
an object or substance; or
a cavity or hole in an object or substance.

### Claim 13 (depends on 11)

A method as recited in claim 11, further comprising:
tracking, by the head-mounted display device, at least one of a head location or head orientation of the user;
wherein dynamically modifying the user voice data is further based on at least one of the head location or head orientation of the user.

### Claim 14 (depends on 13)

A method as recited in claim 13, further comprising:
obtaining a surface reconstruction of the physical environment in which the user is located, wherein dynamically modifying the user voice data is further based on the surface reconstruction.

### Claim 15

A head-mounted display device comprising:
a head fitting by which to mount the head-mounted display device to the head of a user;
a display element, coupled to the head fitting and configured to display to the user a holographic image on a real-world view of an environment of the user, the holographic image depicting a physical thing;
a plurality of microphones to input voice of the user while the holographic image is being displayed to the user;
a processor configured to access the audio data and to dynamically modify user voice data representing the voice of the user in real-time to incorporate an audio effect corresponding to the physical thing, while the holographic image is being displayed to the user; and
a speaker subsystem to output sound to the user based on the modified user voice data while the holographic image is being displayed to the user, the sound representing the voice of the user as affected by the physical thing.

### Claim 16 (depends on 15)

A head-mounted display device as recited in claim 15, wherein the plurality of microphones comprises a plurality of omnidirectional microphones configured in an end-fire array.

### Claim 17 (depends on 15)

A head-mounted display device as recited in claim 15, wherein the physical thing is at least one of:
an object or substance; or
a cavity or hole in an object or substance.

### Claim 18 (depends on 15)

A head-mounted display device as recited in claim 15, further comprising:
a head-tracking subsystem to track a current head location and head orientation of the user; and
a surface mapping subsystem to identify locations of physical surfaces in the environment of the user and to generate a surface reconstruction based on the identified locations of physical surfaces;
wherein the processor is configured to dynamically modify the user voice data further based on at least one of the head location or head orientation of the user.

### Claim 19 (depends on 18)

A head-mounted display device as recited in claim 18, wherein the processor is configured to modify the user voice data based on the surface reconstruction.

### Claim 20 (depends on 18)

A head-mounted display device as recited in claim 18, wherein dynamically modifying the user voice data comprises determining at least one of:
a directionality parameter of the modified user voice data, based on at least one of the head location or head orientation of the user;
a volume parameter of the modified user voice data, based on at least one of the head location or head orientation of the user; or
a reverberation parameter of the modified user voice data, based on at least one of the head location or head orientation of the user.
