# US 2019/0377538 A1 · Information Presentation Through Ambient Sounds

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377538 A1
- Document ID: US20190377538A1
- Status: Published application
- Kind: A1
- Application number: 16/007,335
- Filed: 2018-06-13
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: CURIOUS Corporation
- Applicants: Curious Company, LLC
- Inventors: Anthony Mark Jones; Bruce A. Young
- Web page: https://patentgrep.com/patent/US20190377538A1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer system detects an object in a field-of-view (FOV) using at least one sensor coupled to the computer system and determines a shape of the object using a processor of the computer system. The gaze direction of a user of the computer system is determined using at least one sensor coupled to the system. If the object shape intersecting with the current gaze direction has an associated ambient sound that conveys information about the object, the sound is presented using one or speakers coupled to the computer system.

## Claims

### Claim 1

A method for providing sound to a user, the method comprising:
detecting a gaze direction of an eye of the user using gaze detection hardware;
selecting an object based on the gaze direction;
obtaining information related to the object;
choosing a sound based on the information;
retrieving a digital representation of the sound; and
rendering the digital representation of the sound to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the object is hidden from view of the user.

### Claim 3 (depends on 1)

The method of claim 1, wherein the information is a type or class of the object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the information is based on a physical property of the object.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
sensing the physical property of the object using a hardware sensor;
receiving sensor data based on said sensing; and
determining the information related to the object based on the sensor data.

### Claim 6 (depends on 1)

The method of claim 1, wherein the information comprises a hazard related to the object, an identity of something near the object, an identity of something behind the object, an identity of something inside of the object, or an indication of a safe path to the object.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
detecting a predetermined eye gesture performed by the user; and
starting said rendering of the digital representation of the sound in response to said detection of the first predetermined eye gesture.

### Claim 8 (depends on 7)

The method of claim 7, wherein the predetermined eye gesture comprises a change in the gaze direction of the eye of the user from a first position pointing away from the object to a second position pointing at the object.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
detecting a predetermined eye gesture performed by the user; and
halting said rendering of the digital representation of the sound in response to said detection of the second predetermined eye gesture.

### Claim 10 (depends on 9)

The method of claim 9, wherein the predetermined eye gesture comprises a change in the gaze direction of the eye of the user from a second position pointing at the object to a third position pointing away from the object.

### Claim 11 (depends on 1)

The method of claim 1, wherein said rendering of the digital representation of the sound is performed to make the sound non-intrusive to the user.

### Claim 12 (depends on 11)

The method of claim 11, wherein said non-intrusive rendering of the digital representation of the sound is mixed to be a background sound with other sounds.

### Claim 13 (depends on 1)

The method of claim 1, wherein said rendering of the digital representation of the sound is presented to the user as a directional sound originating at the object.

### Claim 14 (depends on 1)

The method of claim 1, wherein an association between the information and the sound is based on a setting provided by the user.

### Claim 15 (depends on 1)

The method of claim 1, wherein the sound is selected to convey the information to the user.

### Claim 16 (depends on 15)

The method of claim 15, wherein the information is not visible to the eye of the user by viewing the object.

### Claim 17

An article of manufacture comprising a tangible medium, that is not a transitory propagating signal, encoding computer-readable instructions that, when applied to a computer system, instruct the computer system to perform a method comprising:
detecting a gaze direction of an eye of a user using gaze detection hardware;
establishing a position of an object in a field of view of the user using a sensor;
defining a boundary around the object;
determining that the gaze direction intersects the boundary around the object;
obtaining information associated with the object;
choosing a sound based on the information; and
playing the sound to the user.

### Claim 18 (depends on 17)

The article of manufacture of claim 17, the method further comprising:
ascertaining a type of the object based an image of the object obtained from the sensor, wherein the information associated with the object comprises the type of the object;
using the type of the object as an index into a table of sounds associated with types of objects.

### Claim 19

A head-mounted display (HMD) comprising:
a display;
a structure, coupled to the display and adapted to position the display in a field-of-view (FOV) of the user;
an eye gaze detection subsystem, coupled to the structure;
a sound reproduction device, coupled to the structure; and
a processor, coupled to the display, the eye gaze detection subsystem, and the sound reproduction device, the processor configured to:
detect a gaze direction of an eye of a wearer of the HMD using the eye gaze detection subsystem;
select an object based on the gaze direction;
obtain information related to the object;
choose a sound based on the information;
retrieve a digital representation of the sound; and
render the digital representation of the sound to the user through the at least one sound reproduction device.

### Claim 20 (depends on 19)

The HMD of claim 19, further comprising a sensor, coupled to the processor;
the processor further configured to:
receive sensor data related to the object from the sensor; and
determine the information related to the object based on the sensor data.
