US 10,884,493 B2Granted

Gesture based user interfaces, apparatuses and systems using eye tracking, head tracking, hand tracking, facial expressions and other user actions

AssigneeUday Parshionikar

InventorUday Parshionikar

Life of this patent

Estimating the term…

Loading patent…

Abstract

User interaction concepts, principles and algorithms for gestures involving facial expressions, motion or orientation of body parts, eye gaze, tightening muscles, mental activity, and other user actions are disclosed. User interaction concepts, principles and algorithms for enabling hands-free and voice-free interaction with electronic devices are disclosed. Apparatuses, systems, computer implementable methods, and non-transient computer storage media storing instructions, implementing the disclosed concepts, principles and algorithms are disclosed. Gestures for systems using eye gaze and head tracking that can be used with augmented, mixed or virtual reality, mobile or desktop computing are disclosed. Use of periods of limited activity and consecutive user actions in orthogonal axes is disclosed. Generation of command signals based on start and end triggers is disclosed. Methods for coarse as well as fine modification of objects are disclosed.

Claims24

Dependent claims

A system for a user to control an electronic device, the system comprising:

at least one sensor configured to provide a portion of at least one of
i) head information (Head Info) indicative of at least one of motion and position of the user's head,
ii) eye gaze information (Eye Info) indicative of at least one of motion and position of the user's eye gaze, and
iii) facial expression (FE) information (FE Info) indicative of at least one of motion or position of one or more parts of the user's face;
and
at least one processor configured to
i) start a first detection step, wherein the first detection step detects at least one of
a. the Head Info reaching or crossing a first head threshold, wherein the first head threshold is an amount of motion of the user's head or a position of the user's head,
b. the Eye Info reaching or crossing a first eye gaze threshold, wherein the first eye gaze threshold is an amount of motion of the user's eye gaze or a position of the user's eye gaze, and
c. a first FE becoming active for at least a first minimum FE detection time, wherein the first FE is said to be active when the FE Info reaches or crosses a first FE threshold and stays at or beyond the first FE threshold,
ii) after successful completion of the first detection step, start generation of command signals based on at least one of the Head Info and the Eye Info,
iii) after the start generation of command signals, start a second detection step, wherein the second detection step detects at least one of
a. the Head Info is within a second head threshold for at least a minimum head period of limited activity (POLA) time duration, wherein the second head threshold is less than the first head threshold and wherein the minimum head POLA time duration is greater than zero,
b. the Eye Info is within a second eye threshold for at least a minimum eye gaze POLA time duration, wherein the second eye gaze threshold is less than the first eye gaze threshold and wherein the minimum eye gaze POLA time duration is greater than zero, and
c. a second FE becoming active for at least a second minimum FE detection time, wherein the second FE is said to be active when the FE Info reaches or crosses a second FE threshold and stays at or beyond the second FE threshold, and wherein the second FE becomes active only after the first FE is no longer active, and
iv) after successful completion of the second detection step, stop generation of the command signals.

13 more elements

A computer implemented method of controlling an electronic device by a user, the computer implemented method comprising:

receiving at least one of
i) head information (Head Info) indicative of at least one of motion and position of the user's head,
ii) eye gaze information (Eye Info) indicative of at least one of motion and position of the user's eye gaze, and
iii) facial expression (FE) information (FE Info) indicative of at least one of motion or position of one or more parts of the user's face;
monitoring the received information for detection of a first event based on detecting at least one of
i) the Head Info reaching or crossing a first head threshold, wherein the first head threshold is an amount of motion of the user's head or a position of the user's head,
ii) the Eye Info reaching or crossing a first eye gaze threshold, wherein the first eye gaze threshold is an amount of motion of the user's eye gaze or a position of the user's eye gaze, and
iii) a first FE becoming active for at least a first minimum FE detection time, wherein the first FE is said to be active when the FE Info reaches or crosses a first FE threshold and stays at or beyond the first FE threshold;
after successful detection of the first event, start generating command signals based on at least one of the Head Info and the Eye Info;
after the start of generating command signals, start monitoring the received information for detection of second event wherein the second event is detected when at least one of
i) the Head Info is within a second head threshold for at least a minimum head period of limited activity (POLA) time duration, wherein the second head threshold is less than the first head threshold and wherein the minimum head POLA time duration is greater than zero,
ii) the Eye Info is within a second eye threshold for at least a minimum eye gaze POLA time duration, wherein the second eye gaze threshold is less than the first eye gaze threshold and wherein the minimum eye gaze POLA time duration is greater than zero, and
iii) a second FE becoming active for at least a second minimum FE detection time, wherein the second FE is said to be active when the FE info reaches or crosses a second FE threshold and stays at or beyond the second FE threshold, and wherein the second FE becomes active only after the first FE is no longer active;
and
after detection of the end trigger, stop generating the command signals.

12 more elements

A non-transitory computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, said one or more programs causing performance of a method comprising:

receiving at least one of
i) head information (Head Info) indicative of at least one of motion and position of the user's head,
ii) eye gaze information (Eye Info) indicative of at least one of motion and position of the user's eye gaze, and
iii) facial expression (FE) information (FE Info) indicative of at least one of motion or position of one or more parts of the user's face;
detecting a first event based on at least one of
i) the Head Info reaching or crossing a first head threshold, wherein the first head threshold is an amount of motion of the user's head or a position of the user's head,
ii) the Eye Info reaching or crossing a first eye gaze threshold, wherein the first eye gaze threshold is an amount of motion of the user's eye gaze or a position of the user's eye gaze, and
iii) a first FE becoming active for at least a first minimum FE detection time, wherein the first FE is said to be active when the FE info reaches or crosses a first FE threshold and stays at or beyond the first FE threshold;
after successful detection of the first event, start generating command signals based on at least one of the Head Info and the Eye Info;
after the start generating of command signals, start detection of second event wherein the second event is detected when at least one of
i) the Head Info is within a second head threshold for at least a minimum head period of limited activity (POLA) time duration, wherein the second head threshold is less than the first head threshold and wherein the minimum head POLA time duration is greater than zero,
ii) the Eye Info is within a second eye threshold for at least a minimum eye gaze POLA time duration, wherein the second eye gaze threshold is less than the first eye gaze threshold and wherein the minimum eye gaze POLA time duration is greater than zero, and
iii) a second FE is becoming active for at least a second minimum FE detection time, wherein the second FE is said to be active when the FE info reaches or crosses a second FE threshold and stays at or beyond the second FE threshold, and wherein the second FE becomes active only after the first active FE is no longer active;
and
after successful detection of the second event, stop the generating of the command signals.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/726,350Filed 24 Dec 2019 · continuation in part
Priority: 20 Jun 2013
Published: US 2020/0249752 A16 Aug 2020
Granted: 5 Jan 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification20 codes

G06F 3/013
Arrangements for interaction with the human body, e.g. for user immersion in virtual reality: eye tracking input arrangements
G06F 1/163
For portable computers: wearable computers, e.g. on a belt
G06F 1/1686
Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675: the I/O peripheral…
G06F 1/1694
Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675: the I/O peripheral…

Family

Application61/837,215Us provisional application · filed 20 Jun 2013
PCT/US2014/043529Parent of this continuation-in-part · filed 20 Jun 2014
Application62/313,042Us provisional application · filed 24 Mar 2016
Application62/427,006Us provisional application · filed 28 Nov 2016
Application62/470,872Us provisional application · filed 14 Mar 2017
Application15/469,456Parent of this continuation-in-part · filed 24 Mar 2017
Application62/537,482Us provisional application · filed 27 Jul 2017
Application62/589,228Us provisional application · filed 21 Nov 2017
Application62/626,253Us provisional application · filed 5 Feb 2018
Application62/630,253Us provisional application · filed 14 Feb 2018
Application15/921,632Parent of this continuation-in-part · filed 14 Mar 2018
Application16/201,776Parent of this continuation-in-part · filed 27 Nov 2018
US 2020/0249752 A1Pre-grant publication · published 6 Aug 2020
US 10,137,363 B2Parent of this continuation-in-part
Application14/897,657Parent of this continuation-in-part
US 10,254,844 B2Parent of this continuation-in-part
US 10,558,272 B2Parent of this continuation-in-part
US 2016/0109961 A1Published application
US 2017/0249009 A1Published application
US 2018/0364810 A1Published application
US 2019/0265802 A1Published application
US 10,884,493 B2This patent

From the US filing history. Foreign family members aren’t listed.