# US 2025/0165066 A1 · Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165066 A1
- Document ID: US20250165066A1
- Status: Published application
- Kind: A1
- Application number: 18/966,049
- Filed: 2024-12-02
- Published: 2025-05-22
- Earliest claimed: 2015-12-11
- Assignees: Perceptive Devices LLC
- Applicants: Perceptive Devices LLC
- Inventors: Uday Parshionikar
- Web page: https://patentgrep.com/patent/US20250165066A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06V 20/20: in augmented reality scenes
- G06V 40/174: {Facial expression recognition}
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F 2203/04801: Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

User interaction concepts, principles and algorithms for gestures involving facial expressions, motion or orientation of body parts, eye gaze, tightening muscles, mental activity, and other user actions are disclosed. User interaction concepts, principles and algorithms for enabling hands-free and voice-free interaction with electronic devices are disclosed. Apparatuses, systems, computer implementable methods, and non-transient computer storage media storing instructions, implementing the disclosed concepts, principles and algorithms are disclosed. Gestures for systems using eye gaze and head tracking that can be used with augmented, mixed or virtual reality, mobile or desktop computing are disclosed. Use of periods of limited activity and consecutive user actions in orthogonal axes is disclosed. Generation of command signals based on start and end triggers is disclosed. Methods for coarse as well as fine modification of objects are disclosed.

## Claims

### Claim 1

A system for a user to control an electronic device, the system comprising:
one or more first sensors configured to provide eye gaze information (Eye Info) indicative of the user's eye gaze;
one or more second sensors configured to provided facial action (FA) information (FA Info) indicative of one or more facial actions of the user; and
one or more processors configured to
i) receive said Eye Info and said FA Info;
ii) perform a first detection step utilizing said FA Info to detect a first facial action being performed by the user; and
iii) upon detection of said first facial action, generate one or more first command signals based on said Eye Info;
wherein said one or more first command signals are generated as a one-time output; and
wherein said one or more first command signals modify an object of interest (OOI).

### Claim 2 (depends on 1)

The system of claim 1, wherein said first facial action comprises blinking one or more eyes of the user.

### Claim 3 (depends on 1)

The system of claim 1, wherein said first facial action comprises one or more eyelids or eyebrows of the user moving from a first position to a second position, followed by a return to a position substantially similar to said first position, wherein said first position and said second position are distinct.

### Claim 4 (depends on 1)

The system of claim 1, wherein said first facial action comprises at least one of chin, nose, mouth, a lip, or a cheek of the user moving from a third position to a fourth position, followed by a return to a position substantially similar to said third position, wherein said third position and said fourth position are distinct.

### Claim 5 (depends on 1)

The system of claim 1, wherein said first command signals are based on Eye Info received before completion of said first detection step.

### Claim 6 (depends on 1)

The system of claim 1, wherein said first command signals are based on Eye Info received during said first detection step.

### Claim 7 (depends on 1)

The system of claim 1, wherein said first command signals are based on Eye Info received after completion of said first detection step.

### Claim 8 (depends on 1)

The system of claim 1, wherein said first command signals are based on Eye Info received at least one of: before, during, or after said successful completion of said first detection step.

### Claim 9 (depends on 1)

The system of claim 1, the system further comprising:
one or more third sensors configured to provide a portion of head information (Head Info) indicative of at least one of motion or position of the user's head;
wherein said one or more processors are further configured to
i) receive said Head Info;
ii) immediately after generating said one or more first command signals, start a second detection step utilizing said FA Info, wherein said one or more processors detect a second facial action being performed by the user;
iii) immediately after generating said one or more first command signals, start generating one or more second command signals based on at least one of said Eye Info or said Head Info; and
iv) upon detection of said second facial action, end generating said one or more second command signals.

### Claim 10 (depends on 9)

The system of claim 9, wherein said one or more processors are further configured to
i) generate said second command signals based on Eye Info and not Head Info, when said Eye Info is greater than or equal to an Eye Info threshold; and
ii) generate said second command signals on Head Info and not Eye Info, when said Eye Info is less than said Eye Info threshold.

### Claim 11 (depends on 1)

The system of claim 1, wherein the system further comprises a display mechanism, wherein the display mechanism displays said OOI.

### Claim 12 (depends on 11)

The system of claim 11, wherein said display mechanism is mounted on the user's head.

### Claim 13 (depends on 1)

The system of claim 1, wherein said first command signals modify at least one of location or appearance of said OOI.

### Claim 14 (depends on 1)

The system of claim 1, wherein at least one of said one or more first sensors is worn on the body of the user.

### Claim 15

A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:
receiving eye gaze information (Eye Info) indicative of the user's eye gaze and facial action (FA) information (FA Info) indicative of one or more facial actions of the user;
performing a first detection step utilizing said FA Info to detect a first facial action being performed by the user; and
upon detection of said first facial action, generating first command signals based on said Eye Info;
wherein said one or more first command signals are generated as a one-time output; and
wherein said one or more first command signals modify an object of interest (OOI).

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first facial action comprises blinking one or more eyes of the user.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first facial action comprises one or more eyelids or eyebrows of the user moving from a first position to a second position, followed by a return to a position substantially similar to said first position, wherein said first position and said second position are distinct.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first facial action comprises at least one of chin, nose, mouth, a lip, or a cheek of the user moving from a third position to a fourth position, followed by a return to a position substantially similar to said third position, wherein said third position and said fourth position are distinct.

### Claim 19 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first command signals are based on Eye Info received before completion of said first detection step.

### Claim 20 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first command signals are based on Eye Info received during said first detection step.

### Claim 21 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first command signals are based on Eye Info received subsequent to completion of said first detection step.

### Claim 22 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said first command signals are based on Eye Info received at least one of: before, during, or subsequent to said successful completion of said first detection step.

### Claim 23 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein said method further comprises:
receiving of head information (Head Info) indicative of at least one of motion or position of the user's head;
immediately after generating said one or more first command signals, start a second detection step utilizing said FA Info to detect a second facial action being performed by the user;
immediately after generating said one or more first command signals, start generating one or more second command signals based on at least one of said Eye Info or said Head Info; and
upon detection of said second facial action, end generating said one or more second command signals.

### Claim 24 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein said method further comprises:
i) generating said second command signals based on Eye Info and not Head Info, when said Eye Info is greater than or equal to an Eye Info threshold; and
ii) generating said second command signals based on Head Info and not Eye Info, when said Eye Info is less than said Eye Info threshold.

### Claim 25

A head-worn apparatus for a user, the apparatus comprising:
one or more first sensors configured to provide eye gaze information (Eye Info) indicative of the user's eye gaze;
one or more second sensors configured to provide facial action (FA) information (FA Info) indicative of one or more facial actions of the user;
one or more first display mechanisms; and
one or more processors configured to:
i) receive said Eye Info and said FA Info;
ii) perform a first detection step utilizing said FA Info to detect a first facial action being performed by the user; and
iii) upon detection of said first facial action, generate first command signals based on said Eye Info;
wherein said one or more first command signals are generated as a one-time output;
wherein said one or more first command signals modify an object of interest (OOI); and
wherein said OOI is displayed using said one or more first display mechanisms.

### Claim 26 (depends on 25)

The head-worn apparatus of claim 25, wherein said first facial action comprises action performed by the user utilizing at least one of: eyelid muscles or eyebrow muscles.

### Claim 27 (depends on 25)

The head-worn apparatus of claim 25, wherein said first command signals are based on Eye Info received at least one of: before, during, or after completion of said first detection step.

### Claim 28 (depends on 25)

The head-worn apparatus of claim 25, the apparatus further comprising:
one or more third sensors configured to provide head information (Head Info) indicative of at least one of motion or position of the user's head;
wherein said one or more processors are further configured to:
i) receive said Head Info; and
ii) subsequent to generating said first command signals, start generating one or more second command signals based on at least one of: said Eye Info, said Head Info, or said FA Info.

### Claim 29 (depends on 28)

The head-worn apparatus of claim 28, wherein said one or more processors are further configured to
i) generate said second command signals based on Eye Info when said Eye Info is greater than or equal to an Eye Info threshold; and
ii) generate said second command signals based on Head Info when said Eye Info is less than said Eye Info threshold.

### Claim 30 (depends on 25)

The head-worn apparatus of claim 25, wherein said one or more first display mechanisms comprise a display screen or a retinal projector.

### Claim 31 (depends on 25)

The head-worn apparatus of claim 25, wherein said first facial action comprises blinking one or more eyes of the user.

### Claim 32 (depends on 25)

The head-worn apparatus of claim 25, wherein said first facial action comprises one or more eyelids or eyebrows of the user moving from a first position to a second position, followed by a return to a position substantially similar to said first position, wherein said first position and said second position are distinct.

### Claim 33 (depends on 25)

The head-worn apparatus of claim 25, wherein said first facial action comprises at least one of chin, nose, mouth, a lip, or a cheek of the user moving from a third position to a fourth position, followed by a return to a position substantially similar to said third position, wherein said third position and said fourth position are distinct.
