# US 12,118,190 B2 · Method for processing data to adjust data inputting speed in human-computer interface system controlled by eye gaze and electroencephalography data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,190 B2
- Document ID: US12118190B2
- Status: Granted
- Kind: B2
- Application number: 18/210,342
- Filed: 2023-06-15
- Published: 2023-10-12
- Granted: 2024-10-15
- Earliest claimed: 2023-06-15
- Assignees: University of Engineering and Technology - Vietnam National University
- Applicants: Ha Thanh Le; Dang Hai Kieu; Hoa Minh Nguyen; Anh Viet Nguyen; Duyen Thi Ngo; Hung Ba Nguyen; Khanh Quoc Man
- Inventors: Ha Thanh Le; Dang Hai Kieu; Hoa Minh Nguyen; Anh Viet Nguyen; Duyen Thi Ngo; Hung Ba Nguyen; Khanh Quoc Man
- Web page: https://patentgrep.com/patent/US12118190B2

## CPC

- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present invention relates to a method for processing data executed by a computing apparatus including at least a storage and a processor in a human-computer interface system using electroencephalography (EEG) data and eye tracking (ET) data, the method comprising following steps: representing through the control interface, a set of list of keys which is associated with user actions; collecting and processing the two EEG and ET dataset, to determine whether the user is looking at a certain key or not and the concentration state of the user while looking at that key, if yes then activating a time counting parameter t with an increasing rate of the time counting parameter Δ>0 and an adjusting rate ΔT; confirming whether the user wants to select that key or not, based on a time constraint defined as t≥dwell, wherein dwell is a known parameter according to a conventional method corresponding to a required time duration that the user has to look at a key to select that key. The present invention provides a method for adjusting increase or decrease the time counting parameter t via the adjusting rate ΔT which is determined based on the concentration state, to establish an actual time duration in which the user is capable to enter the key faster or slower than the known parameter dwell, adjusting faster or slower according to the concentration state of the user, thus the key inputting using eye gaze becomes more accurate and realistic.

## Claims

### Claim 1

A method for processing data executed by a computing apparatus with at least a storage and a processor in an eye gaze controlled human-computer interface system using electroencephalography (EEG) data and eye tracking (ET) data,
wherein the human-computer interface system includes at least a screen configured to display a control interface for user via a list of keys which is associated with user actions, an electroencephalography monitoring device for collecting the electroencephalography data, and an eye tracking device for collecting the eye tracking data,
the method comprising following steps:
representing through the control interface, at least one or more sets of the list of keys which is associated with user actions in a specific context;
collecting and preprocessing an ET dataset while the user is looking at the control interface to determine the eye tracking data which is data of coordinate form (x,y) corresponding to the position on the screen that the user's eyes look at;
defining the key that the user wants to select based on the (x,y) coordinates on the screen that the user's eyes be looking at by calculating whether said (x,y) coordinates is within the boundary of a certain key or not;
setting up and calculating a time counting parameter t, during the (x,y) coordinates is within the boundary of the key defined in the foregoing step, to confirm whether the user is trying to select said key or not, based on at least a time constraint defined as t≥dwell,
wherein dwell is a known parameter corresponding to a required time duration that the user has to look at a key to select that key, and when the time constraint is satisfied then confirming that the user wants to select that key,
said setting up and calculating the time counting parameter t are executed as following:
setting up a time counting parameter t=0, an increasing rate of the time counting parameter Δ>0, and an adjusting rate ΔT=0,
performing a loop cycle including:
updating the time counting parameter t according to the formula:
t:=t+Δ+ΔT,
checking the time constraint t≥dwell, and
updating the adjusting rate ΔT based on the EEG data,
wherein said updating the adjusting rate ΔT is carried out as follow:
collecting the EEG data within about n seconds right before the time t of the current loop that is the time duration from t−n to t;
preprocessing the obtained EEG data;
extracting features of the preprocessed EEG dataset;
inputting the extracted features from the EEG data into the classifier M to determine a concentration state p and a confidence score confp; and
setting up the increasing rate of the time counting parameter ΔT based on the concentration state p and the confidence score confp.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the list of keys are keys, buttons, control keys, characters, tags, icons, or a combination thereof.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the receiving time of the EEG data since the user starts looking at a key that the user wants to “press” n selected within about 0.1 to 0.45 seconds.

### Claim 4 (depends on 1)

The method according to claim 1, wherein dwell selected within about 0.3 to 3 seconds.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the increasing rate of the time counting parameter is determined by the formula Δ=1/nc seconds, wherein nc is the number of cycles that the system executes in one second and nc selected within about 100 to 2000.

### Claim 6 (depends on 1)

The method according to claim 1, wherein extracting the features of the EEG dataset is extracting the CSP (Common Spatial Pattern) features sif the EEG data received from the electroencephalography monitoring device with the number of channels less than the predetermined number of channels c or extracting the PSD (Power Spectral Density) features in vice versa, wherein c selected within about 14 to 64.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the classifier M using a support vector machine model, a multilayer perceptron model, or a same classification model.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the user experience coefficients βp selected within about 0 to 3 and θ selected within about 0.5 to 0.8.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the adjusting rate ΔT is updated according to the formula:
Δ
T
=
{
(
p
*
c
⁢
o
⁢
n
⁢
f
p
*
β
p
)
/
nc
if
⁢
conf
p
≥
θ
0
in
⁢
vive
⁢
versa
,
wherein p is a label indicating the concentration state (p=−1 means not concentration and p=+1 means concentration) and confp∈[0,1] is the confidence score of the label (the greater confp the more confident the label p), p and confp are returned from the classifier M, and βp is a user experience coefficient.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the electroencephalography monitoring device (EEG) is a wearable headset or a cap which includes plurality of sensors in form of contact electrodes on the user's head.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the eye tracking device is a fixed device embedded in the screen, a device connected or attached on the screen of an electronic apparatus, or a device which is wearable by the user preferred as a glasses.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the eye tracking device comprising an infrared camera.

### Claim 13

A method for processing data executed by computing apparatus including at least a storage and a processor in an eye gaze controlled human-computer interface system using electroencephalography (EEG) data and eye tracking (ET) data,
wherein the human-computer interface system includes at least a screen configured to display a control interface for user via a list of keys which is associated with user actions, an electroencephalography monitoring device for collecting the electroencephalography data, and an eye tracking device for collecting the eye tracking data,
the method comprising following steps:
representing through the control interface, at least one or more sets of the list of keys which is associated with user actions in a specific context;
collecting and preprocessing an ET dataset while the user is looking at the control interface to determine the eye tracking data which is data of coordinate form (x,y) corresponding to the position on the screen that the user's eyes look at;
defining the key that the user wants to select based on the (x,y) coordinates on the screen that the user's eyes be looking at by calculating whether said (x,y) coordinates is within the boundary of a certain key or not;
setting up and calculating a time counting parameter t, during the (x,y) coordinates be within the boundary of the key defined in the foregoing step, to confirm whether the user is trying to select said key or not, based on at least a time constraint defined as t≥dwell,
wherein dwell is a known parameter corresponding to a required time duration that the user has to look at a key to select that key, and when the time constraint is satisfied then confirming that the user wants to select that key,
said setting up and calculating the time counting parameter t are executed as following:
setting up a time counting parameter t=0, an increasing rate of the time counting parameter Δ>0, an adjusting rate ΔT=0,
performing a loop cycle including:
updating the time counting parameter t according to the formula:
t:=(t+Δ+ΔT)*βcus,
wherein βcus is a competent using skill coefficient which is determined to be proportional to the level of the competent using skill of user,
checking the time constraint t≥dwell, and
updating the adjusting rate ΔT based on the EEG data,
wherein said updating the adjusting rate ΔT is carried out as follow:
collecting the EEG data within about n seconds right before the time t of the current loop that is the time duration from t−n to t;
preprocessing the obtained EEG data;
extracting features of the preprocessed EEG dataset;
inputting the extracted features from the EEG data into the classifier M to determine a concentration state p and a confidence score confp; and
setting up the increasing rate of the time counting parameter ΔT based on the concentration state p and the confidence score confp.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the competent using skill coefficient βcus having the default value is 1.

### Claim 15 (depends on 13)

The method according to claim 13, wherein the competent using skill coefficient βcus adjusted to increase as the level of the competent using skill of the user increases.