# US 10,321,258 B2 · Emulating spatial perception using virtual echolocation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,321,258 B2
- Document ID: US10321258B2
- Status: Granted
- Kind: B2
- Application number: 15/491,897
- Filed: 2017-04-19
- Published: 2018-10-25
- Granted: 2019-06-11
- Earliest claimed: 2017-04-19
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Brandon Myles Arteaga; Wilson Jacob Dreewes; Jeffrey Michael Davis; Zainuddin Qazi; Boris Eduardo Baracaldo Doikova
- Web page: https://patentgrep.com/patent/US10321258B2

## CPC

- H04S 7/304: {For headphones}
- A61F 9/08: Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
- G01S 15/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S15/46)
- G01S 15/88: Sonar systems specially adapted for specific applications (seismic or acoustic prospecting or detecting G01V1/00)
- G06F 3/012: {Head tracking input arrangements}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09B 21/006: {using audible presentation of the information}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04R 5/0335: {Earpiece support, e.g. headbands or neckrests (for monophonic headphones H04R1/105)}
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- H04S: STEREOPHONIC SYSTEMS
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

Examples are disclosed that relate to a head-mounted device configured to perform virtual echolocation. The head-mounted device is configured to cast an array of rays at specified angles from a position derived from a pose of the head-mounted device in a physical environment, identify a plurality of intersection points of the rays with a virtual model of the physical environment, for each identified intersection point, modify an audio signal based on a head-related transfer function corresponding to the intersection point to produce a plurality of spatialized audio signals, for each spatialized audio signal, determine a time-of-flight adjustment based upon a distance between the corresponding intersection point and the position from which the rays were cast, and output each spatialized audio signal to one or more speakers with a delay based on the time-of-flight adjustment.

## Claims

### Claim 1

A head-mounted device, comprising:
a depth camera configured to image a physical environment;
one or more motion sensors configured to measure one or more motion parameters;
a left ear speaker and a right ear speaker;
a processing system; and
a storage system holding instructions executable by the processing system to
generate a virtual model of the physical environment based upon one or more depth images from the depth camera;
determine a pose of the head-mounted device in the physical environment based upon the one or more motion parameters from the one or more motion sensors;
cast an array of rays at specified angles from a position derived from the pose of the head-mounted device;
identify a plurality of intersection points of the rays with the virtual model;
for each intersection point of the plurality of intersection points, modify an input audio signal based on a head related transfer function (HRTF) corresponding to the intersection point to produce a spatialized audio signal corresponding to the intersection point;
for each spatialized audio signal of a plurality of spatialized audio signals corresponding to the plurality of intersection points, determine a time-of-flight adjustment based upon a distance between the corresponding intersection point and the position from which the rays were cast;
for each spatialized audio signal of the plurality of spatialized audio signals, output the spatialized audio signal to one or more of the left ear speaker and the right ear speaker with a delay corresponding to the time-of-flight adjustment for the spatialized audio signal; and
repeatedly output a plurality of updated spatialized audio signals to the left ear and/or right ear speakers according to a refresh rate, wherein the plurality of updated spatialized audio signals are based upon an updated pose of the head-mounted device.

### Claim 2 (depends on 1)

The head-mounted device of claim 1, wherein the storage system further holds instructions executable by the processing system to modify the input audio signal by one or more of changing a volume of the input audio signal to account for a position in the physical environment corresponding to the intersection point from which the spatialized audio signal emanates and changing a directionality of the audio signal to account for a position in the physical environment corresponding to the intersection point from which the spatialized audio signal emanates.

### Claim 3 (depends on 1)

The head-mounted device of claim 1, wherein the storage system further holds instructions executable by the processing system to dynamically adjust the refresh rate based upon receiving user input from a wearer of the head-mounted device.

### Claim 4 (depends on 1)

The head-mounted device of claim 1, wherein the storage system further holds instructions executable by the processing system to dynamically adjust the refresh rate based upon the one or more motion parameters from the one or more motion sensors.

### Claim 5 (depends on 1)

The head-mounted device of claim 1, wherein the storage system further holds instructions executable by the processing system to dynamically adjust the specified angles of the rays in the array such that the rays intersect a different region of interest of the virtual model.

### Claim 6 (depends on 5)

The head-mounted device of claim 5, wherein the storage system further holds instructions executable by the processing system to dynamically adjust the specified angles of the rays in the array based upon one or more of a position of a detected object in the physical environment nearest to the pose of the head-mounted device, a pose vector of the head-mounted device, and the one or more motion parameters from the motion sensors.

### Claim 7 (depends on 5)

The head-mounted device of claim 5, wherein the storage system further holds instructions executable by the processing system to dynamically adjust the specified angles of the rays in the array based upon user input received from a wearer of the head-mounted device.

### Claim 8 (depends on 1)

The head-mounted device of claim 1, wherein the audio signal is a first audio signal, and wherein the storage system further holds instructions executable by the processing system to
cast one or more additional rays each having a specified angle and a specified length from the position;
identify one or more additional intersection points of the one or more additional rays with the virtual model;
for each of the one or more additional intersection points, modify a second, different audio signal based on a head related transfer function (HRTF) corresponding to the additional intersection point to produce a second spatialized audio signal corresponding to the intersection point;
for each of one or more second spatialized audio signals corresponding to the one or more additional identified intersection points, determine a time-of-flight adjustment based upon a distance between the corresponding additional intersection point and the position; and
for each of the one or more second spatialized audio signals, output the second spatialized audio signal to one or more of the left ear speaker and the right ear speaker with a delay corresponding to the time-of-flight adjustment for the spatialized audio signal.

### Claim 9

A method for performing virtual echolocation using a computing device in communication with a left ear speaker, and a right ear speaker, the method comprising:
generating a virtual model of an environment;
determining a pose in the environment;
casting an array of rays at specified angles from a position derived from the pose;
identifying a plurality of intersection points of the rays with the virtual model;
for each intersection point of the plurality of intersection points, modifying an input audio signal based on a head related transfer function (HRTF) corresponding to the intersection point to produce a spatialized audio signal corresponding to the intersection point;
for each spatialized audio signal of a plurality of spatialized audio signals corresponding to the plurality of intersection points, determining a time-of-flight adjustment based upon a distance between the corresponding intersection point and the position from which the rays were cast;
for each spatialized audio signal of the plurality of spatialized audio signals, outputting the spatialized audio signal to one or more of the left ear speaker and the right ear speaker with a delay corresponding to the time-of-flight adjustment for the spatialized audio signal; and
repeatedly outputting a plurality of updated spatialized audio signals to the left ear and/or right ear speakers according to a refresh rate, wherein the plurality of updated spatialized audio signals are based upon an updated pose in the environment.

### Claim 10 (depends on 9)

The method of claim 9, wherein the environment is a physical environment, wherein the computing device further includes a depth camera configured to image the physical environment and one or more motion sensors configured to measure one or more motion parameters, wherein the virtual model of the physical environment is generated based upon one or more depth images from the depth camera, and wherein the pose is determined based upon the one or more motion parameters of the one or more motion sensors.

### Claim 11 (depends on 9)

The method of claim 9, wherein the environment is a virtual environment, and wherein the pose is a virtual position in the virtual model of the virtual environment.

### Claim 12 (depends on 9)

The method of claim 9, further comprising:
dynamically adjusting the refresh rate based upon one or more of user input received from a user of the computing device and the one or more motion parameters from the one or more motion sensors.

### Claim 13 (depends on 9)

The method of claim 9, further comprising:
dynamically adjusting the specified angles of the rays in the array such that the rays intersect a different region of interest of the virtual model.

### Claim 14 (depends on 13)

The method of claim 13, wherein the specified angles of the rays in the array are dynamically adjusted based upon one or more of a position of a detected object in the physical environment nearest to the pose, a pose vector, one or more motion parameters from one or more motion sensors of the computing device, and user input received from a user of the computing device.

### Claim 15

A head-mounted device, comprising:
a depth camera configured to image a physical environment;
one or more motion sensors configured to measure one or more motion parameters;
a left ear speaker and a right ear speaker;
a processing system; and
a storage system holding instructions executable by the processing system to:
generate a virtual model of the physical environment based upon one or more depth images from the depth camera;
determine a pose of the head-mounted device in the physical environment based upon the one or more motion parameters from the one or more motion sensors;
cast an array of rays at specified angles from a position derived from the pose of the head-mounted device, wherein the specified angles are dynamically adjustable based upon one or more of user input received from the wearer and the one or more motion parameters from the one or more motion sensors;
identify a plurality of intersection points of the rays with the virtual model;
for each intersection point of the plurality of intersection points, modify an input audio signal based on a head related transfer function (HRTF) corresponding to the intersection point to produce a spatialized audio signal corresponding to the intersection point;
for each spatialized audio signal of a plurality of spatialized audio signals corresponding to the plurality of intersection points, determine a time-of-flight adjustment based upon a distance between the corresponding intersection point and the position from which the rays were cast; and
for each spatialized audio signal of the plurality of spatialized audio signals, output the spatialized audio signal to one or more of the left ear speaker and the right ear speaker with a delay corresponding to the time-of-flight adjustment for the spatialized audio signal.

### Claim 16 (depends on 15)

The head-mounted device of claim 15, wherein the storage system further holds instructions executable by the processing system to repeatedly output a plurality of updated spatialized audio signals to the left ear and/or right ear speakers according to a refresh rate, wherein the plurality of updated spatialized audio signals are based upon an updated pose of the head-mounted device.

### Claim 17 (depends on 15)

The head-mounted device of claim 15, wherein the storage system further holds instructions executable by the processing system to dynamically adjust the refresh rate based upon one or more of receiving user input from a wearer of the head-mounted device and the one or more motion parameters from the one or more motion sensors.

### Claim 18 (depends on 15)

The head-mounted device of claim 15, wherein the audio signal is a first audio signal, and wherein the storage system further holds instructions executable by the processing system to
cast one or more additional rays each having a specified angle and a specified length from the position;
identify one or more additional intersection points of the one or more additional rays with the virtual model;
for each of the one or more additional intersection points, modify a second, different audio signal based on a head related transfer function (HRTF) corresponding to the additional intersection point to produce a second spatialized audio signal corresponding to the intersection point;
for each of one or more second spatialized audio signals corresponding to the one or more additional identified intersection points, determine a time-of-flight adjustment based upon a distance between the corresponding additional intersection point and the position; and
for each of the one or more second spatialized audio signals, output the second spatialized audio signal to one or more of the left ear speaker and the right ear speaker with a delay corresponding to the time-of-flight adjustment for the spatialized audio signal.
