# US 2014/0351770 A1 · Method and apparatus for immersive system interfacing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0351770 A1
- Document ID: US20140351770A1
- Status: Published application
- Kind: A1
- Application number: 13/902,740
- Filed: 2013-05-24
- Published: 2014-11-27
- Earliest claimed: 2013-05-24
- Assignees: Atheer, Inc.
- Applicants: Atheer, Inc.
- Inventors: Nathan Abercrombie
- Web page: https://patentgrep.com/patent/US20140351770A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed are methods and systems for immersive system interfacing. A stimulus is defined in the form of a gesture or posture with an end-effector such as a hand. The stimulus is sensed within an environment and communicated to a processor. In response to the stimulus the processor generates a user interface, the interface being outputted near the end-effector. The stimulus may be identical for all times, all places, and all conditions in the environment. The interface may be generated and outputted in response to the stimulus for all times, places, and conditions in the environment. The apparatus includes a processor adapted to define a stimulus in the form of an end-effector gesture or posture, and to generate a user interface in response to the stimulus. A sensor is adapted to sense the stimulus, and a display is adapted to output the interface to an environment.

## Claims

### Claim 1

A method comprising:
defining, in a processor, a stimulus, said stimulus comprising executing at least one of an end-effector gesture or an end-effector posture;
sensing said stimulus in an environment;
communicating said stimulus to said processor;
in response to said stimulus, generating with said processor a user interface; and
outputting said user interface within said environment proximate a location of said stimulus.

### Claim 2 (depends on 1)

The method of claim 1, wherein said environment is a three dimensional environment.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
said environment comprises a real-world environment.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
said environment comprises an augmented reality environment.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
said environment comprises a virtual reality environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
said stimulus comprises executing at least one of a hand gesture or a hand posture.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
said stimulus comprises executing a configuration of a hand in said environment wherein at least three digits of said hand are substantially extended.

### Claim 8 (depends on 7)

The method of claim 7, wherein:
said configuration comprises at least a thumb, a first finger, and a second finger of said hand being substantially extended.

### Claim 9 (depends on 7)

The method of claim 7 wherein:
sensing said stimulus comprises distinguishing said at least three digits of said hand.

### Claim 10 (depends on 7)

The method of claim 7, comprising:
sensing said stimulus optically.

### Claim 11 (depends on 7)

The method of claim 7 wherein:
said configuration comprises said at least three digits being at least partially spread.

### Claim 12 (depends on 7)

The method of claim 7, wherein:
said configuration defines at least two gaps of at least partial spreading between at least three groups of at least one of said digits.

### Claim 13 (depends on 7)

The method of claim 7 wherein:
said stimulus comprises executing said configuration as a substantially static posture.

### Claim 14 (depends on 7)

The method of claim 7 wherein:
said stimulus comprises executing said configuration as a dynamic gesture.

### Claim 15 (depends on 14)

The method of claim 14, wherein:
said stimulus comprises transitioning from a first configuration of said hand to a second configuration of said hand wherein said at least three digits of said hand are substantially extended.

### Claim 16 (depends on 14)

The method of claim 14, wherein:
said stimulus comprises transitioning from a first configuration of said hand wherein at least a thumb, a first finger, and a second finger of said hand are substantially clenched together to a second configuration of said hand wherein at least said thumb, said first finger, and said second finger of said hand are substantially extended.

### Claim 17 (depends on 14)

The method of claim 14, wherein:
said stimulus comprises transitioning from a first configuration of said hand wherein at least a thumb, a first finger, and a second finger of said hand are pinched together so as to be in contact at tips thereof to a second configuration of said hand wherein at least said thumb, said first finger, and said second finger of said hand are substantially extended.

### Claim 18 (depends on 7)

The method of claim 7 further comprising:
outputting said user interface so as to be substantially aligned with a plane defined by tips of said digits in said configuration.

### Claim 19 (depends on 7)

The method of claim 7, wherein:
said user interface is substantially aligned with a plane defined by a palm of said hand in said configuration.

### Claim 20 (depends on 7)

The method of claim 7 further comprising:
outputting said user interface in front of said hand.

### Claim 21 (depends on 7)

The method of claim 7 further comprising:
outputting at least a portion of said user interface so as to be in contact with at least one of said at least three digits.

### Claim 22 (depends on 1)

The method of claim 1, wherein:
said user interface comprises at least one manipulation site.

### Claim 23 (depends on 22)

The method of claim 22 further comprising:
defining in said processor an input manipulation for said at least one manipulation site;
defining in said processor an input for said at least one manipulation site, such that when said input manipulation is applied to said at least one manipulation site said input is applied to said processor.

### Claim 24 (depends on 22)

The method of claim 22, wherein:
said at least one manipulation site comprises a contact site.

### Claim 25 (depends on 22)

The method of claim 22, wherein:
said user interface comprises a graphical user interface.

### Claim 26 (depends on 22)

The method of claim 22, wherein:
said user interface comprises visible markings corresponding with said at least one manipulation site.

### Claim 27 (depends on 22)

The method of claim 22, wherein:
said at least one manipulation site is one of a plurality of manipulation sites.

### Claim 28 (depends on 27)

The method of claim 27 further comprising:
outputting at least some of said manipulation sites disposed substantially in at least a partial ring substantially circumscribing at least a portion of said hand.

### Claim 29 (depends on 27)

The method of claim 27 further comprising:
outputting at least some of said manipulation sites disposed substantially adjacent so as to be substantially continuous.

### Claim 30 (depends on 27)

The method of claim 27 further comprising:
outputting at least some of said manipulation sites so as to be discrete.

### Claim 31 (depends on 28)

The method of claim 28 further comprising:
defining in said processor an active condition for said at least one manipulation site, wherein said at least one manipulation site applies said input to said processor when said input manipulation is applied thereto;
defining in said processor an inactive condition for said at least one manipulation site, wherein said at least one manipulation site does not apply said input to said processor when said input manipulation is applied thereto.

### Claim 32 (depends on 31)

The method according to claim 31 further comprising:
defining in said processor a switch manipulation such that applying said switch manipulation to said user interface switches at least one of said at least one manipulation site between said inactive condition and said active condition.

### Claim 33 (depends on 31)

The method according to claim 31 further comprising:
defining in said processor a first switch manipulation, wherein applying said first switch manipulation to said user interface switches at least one of said at least one manipulation site from said active condition to said inactive condition; and
defining in said processor a second switch manipulation, wherein applying said second switch manipulation to said user interface switches at least one of said at least one manipulation site from said inactive condition to said active condition.

### Claim 34 (depends on 31)

The method according to claim 31 further comprising:
outputting said at least one manipulation site when in said inactive condition.

### Claim 35 (depends on 27)

The method of claim 27, wherein:
outputting at least some of said manipulation sites disposed substantially in at least first and second substantially concentric at least partial rings substantially circumscribing at least a portion of said hand.

### Claim 36 (depends on 22)

The method of claim 22 further comprising:
outputting said at least one manipulation site disposed proximate a tip of one of said at least three digits.

### Claim 37 (depends on 1)

The method of claim 1 further comprising:
configuring said user interface based at least in part on at least one of:
a status of said environment,
a content of said environment,
a location within said environment of said interface,
a location of said environment,
a time,
conditions within said environment,
conditions external to said environment,
an identity of a user of said end-effector,
a condition of said user, or
a non-input behavior of said user.

### Claim 38 (depends on 1)

The method of claim 1 further comprising:
generating a content of said interface based at least in part on at least one of:
a status of said environment,
a content of said environment,
a location within said environment of said interface,
a location of said environment,
a time,
conditions within said environment,
conditions external to said environment,
an identity of a user of said end-effector,
a condition of said user, or
a non-input behavior of said user.

### Claim 39 (depends on 1)

The method of claim 1 further comprising:
defining said stimulus to be substantially identical for substantially all times, places, and conditions in said environment.

### Claim 40 (depends on 1)

The method of claim 1 further comprising:
executing said steps of generating and outputting said user interface in response to sensing said stimulus for substantially all times, places, and conditions in said environment.

### Claim 41 (depends on 1)

The method of claim 1 further comprising:
instantiating executable instructions on said processor for implementing at least said steps of defining said stimulus and generating said user interface.

### Claim 42 (depends on 41)

The method of claim 41 further comprising:
instantiating said executable instructions on said processor within an operating system for said processor.

### Claim 43

An apparatus, comprising:
a processor adapted to define a stimulus, said stimulus comprising executing at least one of an end-effector gesture or an end-effector posture;
a display in communication with said processor;
a sensor in communication with said processor, said sensor adapted to sense said stimulus;
said processor further adapted to generate a user interface in response to said stimulus; and
said display adapted to output said user interface within an environment proximate a location of said stimulus.

### Claim 44 (depends on 43)

The apparatus of claim 43, wherein:
said display is adapted to output said environment.

### Claim 45 (depends on 43)

The apparatus of claim 43, wherein:
said display comprises a stereo display.

### Claim 46 (depends on 43)

The apparatus of claim 43, wherein:
said sensor is adapted to sense a three dimensional position of said stimulus.

### Claim 47 (depends on 43)

The apparatus of claim 43, wherein:
said sensor is adapted to sense a three dimensional orientation of said stimulus.

### Claim 48 (depends on 43)

The apparatus of claim 43, wherein:
said sensor comprises a stereo sensor.

### Claim 49 (depends on 43)

The apparatus of claim 43, wherein:
said sensor comprises an image sensor.

### Claim 50 (depends on 43)

The apparatus of claim 43, wherein:
said processor, said display, and said sensor are disposed on a wearable device.

### Claim 51 (depends on 43)

The apparatus of claim 43, wherein:
said processor, said display, and said sensor are disposed on a head mounted display.

### Claim 52

A method comprising:
defining, in a processor, a stimulus for a three dimensional environment, said stimulus comprising executing at least one of (i) a hand gesture comprising at least a thumb, a first finger, and a second finger of said hand being substantially extended and at least partially spread or (ii) a hand posture comprising at least a thumb, a first finger, and a second finger of said hand being substantially extended and at least partially spread, said stimulus being substantially identical for substantially all times, places, and conditions in said three dimensional environment;
optically sensing a position and an orientation of said stimulus in said three dimensional environment;
communicating said stimulus to said processor;
in response to said stimulus:
generating with said processor a graphical user interface (GUI) comprising a plurality of contact manipulation sites, said GUI comprising markings corresponding to said manipulation sites;
defining in said processor an input manipulation for at least one of said content manipulation sites;
defining in said processor an input for said at least one of said content manipulation sites such that when said input manipulation is applied to said at least one of said content manipulation sites said input is applied to said processor;
visibly outputting said GUI proximate said location of said stimulus, wherein:
said GUI is substantially aligned with a plane defined by tips of said thumb, first finger, and said second finger;
said GUI is disposed in front of said hand;
at least a portion of said GUI is in contact with at least one of said tips; and
executing said steps of generating and outputting said GUI in response to sensing said stimulus for substantially all times, places, and conditions in said three dimensional environment.

### Claim 53

An apparatus, comprising:
a body adapted to be worn on a head of a user;
a processor engaged with said body, said processor adapted to define a stimulus for a three dimensional environment, said stimulus comprising executing at least one of a group consisting of an end-effector gesture and an end-effector posture;
first and second visual displays in a stereo configuration engaged with said body and in communication with said processor, said first and second visual displays adapted to output said three dimensional environment, said first and second visual displays being arranged so as to generate stereo output, said first and second visual displays engaged with said body such that said first and second visual displays are disposed proximate and substantially aligned with eyes of said user when said user wears said body;
at least one image sensor in communication with said processor, said sensor adapted to sense said stimulus;
said processor further adapted to generate a graphical user interface in response to said stimulus; and
said first and second visual displays further adapted to output said graphical interface within said three dimensional environment proximate a location of said stimulus.

### Claim 54

An apparatus comprising:
means for defining a stimulus, said stimulus comprising executing at least one of an end-effector gesture or an end-effector posture;
means for sensing said stimulus in a three dimensional environment;
means for communicating said stimulus to said processor;
means for generating with said processor a user interface in response to said stimulus; and
means for outputting said user interface within said three dimensional environment proximate a location of said end-effector.
