# US 11,442,536 B2 · EOG-based method and apparatus for asynchronous character input

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,442,536 B2
- Document ID: US11442536B2
- Status: Granted
- Kind: B2
- Application number: 16/324,926
- Filed: 2017-11-09
- Published: 2021-09-09
- Granted: 2022-09-13
- Earliest claimed: 2016-12-08
- Assignees: South China University of Technology
- Applicants: South China University of Technology
- Inventors: Yuanqing Li; Shenghong He
- Web page: https://patentgrep.com/patent/US11442536B2

## CPC

- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0233: {Character input methods}
- G06F 3/0236: {using selection techniques to select from displayed items}
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06N 5/04: Inference or reasoning models
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06F 2218/12: Classification; Matching
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present invention discloses an EOG-based method and apparatus for asynchronous character input, said method comprising the following steps: displaying a virtual keyboard having a total of N keys on a display, the virtual keyboard flickering in rounds, wherein in each round, all N keys randomly flicker once; when spelling, a user blinks after the flickering of a target key, so as to acquire electro-oculogram signals of the user in real-time; capturing electro-oculogram data within a time period following the flickering of a key from the acquired electro-oculogram signals, and using said electro-oculogram data as an original feature vector of the flickering key; N original feature vectors being generated in each cycle, an electro-oculogram identification method is called to identify the N original feature vectors, and outputting a certain result from 0 to N, wherein 0 indicates that a condition for character input is not met, and 1 to N correspond to the N keys on the virtual keyboard. The present invention ensures that blinking by a user may be detected normally, while at the same time effectively eliminating misjudgments of non-blinking signals, thus increasing the accuracy rate of character input.

## Claims

### Claim 1

An EOG-based method for asynchronous character input, characterized in that: the method comprises the following steps:
displaying a virtual keyboard having a total of N keys on a display, the virtual keyboard flickering in rounds, wherein in each round, all N keys randomly flicker once; when spelling, a user blinks after the flickering of a target key, so that an electro-oculogram acquisition module acquires electro-oculogram signals of the user in real-time; capturing electro-oculogram data within a time period following the flickering of a key from the acquired electro-oculogram signals, and using said electro-oculogram data as an original feature vector of the flickering key; N original feature vectors being generated in each cycle, an electro-oculogram identification algorithm is called to identify the N acquired original feature vectors, and the electro-oculogram identification algorithm will output a certain result from 0 to N, wherein 0 indicates that a condition for character input is not met, and 1 to N correspond to the N keys on the virtual keyboard.

### Claim 2 (depends on 1)

The EOG-based method for asynchronous character input according to claim 1, characterized in that: the method comprises the following steps:
S1: the virtual keyboard flickers in rounds, wherein in each round, all N keys randomly flicker once; when spelling, a user is required to blink after the flickering of a target key, so that the electro-oculogram acquisition module acquires electro-oculogram signals of the user in real-time; electro-oculogram data are captured within a time period of 100-500 ms following the flickering of a key from the acquired electro-oculogram signals, and used as the original feature vector of this key in this round of flickering;
S2: the captured original feature data are pre-processed in sequence including removing baseline drift, removing 50 Hz power frequency interference, and 0.1-30 Hz bandpass filtering;
S3: a first-order difference of the electro-oculogram datum x after the pre-processing in the step S2 is found, which is performed specifically by the
di=xi+1−xi
where i denotes an i-th sampling point and d denotes a differential signal;
S4: N features of D=[d1, d2, . . . , dN] after the difference correspond to N key flickerings in one round; SVM classification and waveform detection are performed to obtain N SVM classification results of S=[s1, s2, . . . , si, . . . , sN] and N waveform detection results of W=[W1, W2, . . . , Wi, . . . , WN], wherein only the first largest M scores are retained for the SVM classification result S with the rest set to zero; the SVM classification result S and the waveform detection result W are multiplied to obtain Rt, where t denotes the t-th round:
Rt=[r1t,r2t, . . . ,rit. . . ,rNt]=[s1·w1,s2·w2, . . . ,si·wi, . . . ,sN·wN]
and Rt is an N-dimensional row vector containing M non-zero values;
S5: Rt is traversed, and if rit>0 does not exist, there is no target output in the current round, and the process returns to the step S1 to continue the detection; if rit>0 exists, the key that satisfies rit>0 is regarded as the candidate target of the current round for further judgment; and
S6: in three consecutive detections (t-th, l-th and t-2-th detections), if no key is detected to be a candidate target twice, then there is no target output in the current round, the output result is 0, and the process returns to the step S1 to continue the detection; otherwise, a key corresponding to the maximum rit+rit-1+rit-3 is determined as the target, and its corresponding operation is performed according to the key, and then the process proceeds to the step S1 to continue the detection.

### Claim 3 (depends on 2)

The EOG-based method for asynchronous character input according to claim 2, characterized in that: the virtual keyboard has a total of 40 keys in 4 rows and 10 columns, and all the keys flicker in a “conditional random” manner that is specifically as follows: with a round taking 1.2 s, each of the 40 keys will flicker once in a round for a duration of 100 ms at an interval of 30 ms between two adjacent flickering keys.

### Claim 4 (depends on 3)

The EOG-based method for asynchronous character input according to claim 3, characterized in that: the “conditional random” flickering manner means that the flickering order of 40 keys is random in each round, but the following two conditions need to be met:
(1-1) any key flickers at an interval of not being less than 600 ms in two consecutive rounds; and
(1-2) when the flickering order of the keys in one round is determined, it is necessary to determine the flickering order of the keys in the previous two rounds, that is, trying to ensure that characters appeared around any character in the previous two rounds do not again appear around this character in the current round.

### Claim 5 (depends on 2)

The EOG-based method for asynchronous character input according to claim 2, characterized in that: the waveform detection in the step S4 comprises the following steps:
the following three conditions are detected for a differential signal Di: a. a trough appears 40-140 ms after the appearance of a peak; b. the peak/trough corresponds to the maximum/minimum point of the entire segment of the signal; and
c. the sum of the energy of the signal from the peak to the trough is greater than a preset threshold P;
when all the three conditions are fulfilled in the detection, the waveform of the key i corresponding to the differential signal Di passes the detection; with Iv, taking a value of 0 or 1, wi=0 indicates that the waveform of the feature di does not pass the detection, and wi=1 indicates that the waveform of the feature di passes the detection.

### Claim 6 (depends on 2)

The EOG-based method for asynchronous character input according to claim 2, characterized in that: the SVM classifier used in the step S4 needs to be trained in advance, and the training process comprises the following steps:
(4-1) the electro-oculogram data are acquired from several trials, wherein each of the trials has a specified target key and is carried out for several rounds, and the N keys randomly flicker once in each round, requiring that a user blinks only when the target key flickers, so that each key generates a feature vector in each round in each trial, and all the feature vectors contain positive and negative samples required to train the SVM classifier;
(4-2) among all the acquired feature vectors, the features corresponding to the target key are regarded as positive sample features and given a label 1, and the features corresponding to the non-target key are regarded asnegative sample features and given a label −1; and
(4-3) the SVM classifier is trained with these labeled positive and negative samples.

### Claim 7 (depends on 2)

The EOG-based method for asynchronous character input according to claim 2, characterized in that: when the virtual keyboard has 40 keys, M takes a value of 5 in the step S4.

### Claim 8

An input apparatus characterized in that: this apparatus comprises an electrode, an electro-oculogram acquisition module, a computer host and a display; the electrode is connected to the electro-oculogram acquisition module through a self-contained connecting wire, an electro-oculogram signal obtained by the electro-oculogram acquisition module is transmitted to the computer host through a common serial port, and the computer host is connected to the display through a common VGA cable;
wherein the display is configured to display a virtual keyboard having a total of N keys on the display, the virtual keyboard is configured to flicker in rounds, and all N keys are configured to flicker randomly once in each round;
wherein the electro-oculogram acquisition module is configured to acquire electro-oculogram signals of a user in real-time through the electrode when the user blinks after the flickering of a target key when spelling, capture electro-oculogram data within a time period following the flickering of a key from the acquired electro-oculogram signals, use said electro-oculogram data as an original feature vector of the flickering key, and generate N original feature vectors in each cycle; and
wherein the computer host is configured to identify the N acquired original feature vectors and output a certain result from 0 to N through an electro-oculogram identification algorithm, wherein 0 indicates that a condition for character input is not met, and 1 to N correspond to the N keys on the virtual keyboard.

### Claim 9 (depends on 8)

The input apparatus according to claim 8, characterized in that: the electro-oculogram acquisition module comprises a signal amplification module, a signal filtering module, an A/D conversion module, and a serial communication module, wherein:
(1) the signal amplification module is based on an INA128 chip, having a common mode rejection ratio of 120 db or more and an amplification factor of 1000 times or more;
(2) the signal filtering module is based on an OPA4227 chip, comprising a second-order Butterworth high-pass filter with a cutoff frequency of 3 Hz, a fourth-order Butterworth low-pass filter with a cutoff frequency of 25 Hz, and a Butterworth 50 Hz trap filter;
(3) the A/D conversion module is based on an AD7606 chip, having a conversion accuracy of 16 bits; and
(4) the serial communication module adopts a serial communication module that comes with STM32.

### Claim 10 (depends on 8)

The input apparatus according to claim 8, characterized in that: the electrode is a common silver chloride electrode.