# US 11,029,758 B2 · Dynamic command remapping for human-computer interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,029,758 B2
- Document ID: US11029758B2
- Status: Granted
- Kind: B2
- Application number: 16/868,348
- Filed: 2020-05-06
- Published: 2021-01-07
- Granted: 2021-06-08
- Earliest claimed: 2019-07-01
- Assignees: Daniel Jonathan Milstein
- Applicants: Daniel Jonathan Milstein
- Inventors: Daniel Jonathan Milstein
- Web page: https://patentgrep.com/patent/US11029758B2

## CPC

- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 3/147: using display panels
- G09G 2380/08: Biomedical applications
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Methods can include: activating a human-computer interface (HCI), such as a brain-computer interface (BCI) or a muscle-computer interface (muCI), through which a computing device is in operative communication with a brain or one or more muscles of a user; defining a command mapping according to which operation of the computing device is configured to be controlled through the HCI based on at least one electric signal of the user measured via one or more sensors coupled to the user; detecting an operation context of the computing device during operation of the computing device; and remapping the command mapping based on the detected operation context during operation of the computing device. The operation of the computing device can then be controlled through the HCI based on the at least one measured electric signal according to the remapped command mapping.

## Claims

### Claim 1

A method comprising:
activating a brain-computer interface (BCI) through which a computing device is in operative communication with a brain of a user;
defining a thought-command mapping according to which operation of the computing device is configured to be controlled through the BCI based on at least one brain signal of the user measured via one or more brain sensors coupled to the brain of the user, the thought-command mapping defining a plurality of thought variants each of which corresponding a command executable by the computing device, a first thought variant among the plurality of thought variants corresponding to a first command executable by the computing device;
detecting an operation context of the computing device during operation of the computing device; and
remapping the thought-command mapping based on the detected operation context during operation of the computing device such that the first thought variant corresponds to a second command executable by the computing device different from the first command,
wherein, in response to the remapping of the thought-command mapping, the operation of the computing device is configured to be controlled through the BCI based on the at least one measured brain signal according to the remapped thought-command mapping, and
wherein the operation context is associated with an application currently executing on the computing device, an external event occurring during operation of the computing device, and a user history associated with the computing device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
measuring the at least one brain signal of the user using the one or more brain sensors;
sensing the first thought variant based on a signal analysis of the at least one measured brain signal;
determining that the second command corresponds to the first thought variant according to the remapped thought-command mapping; and
controlling the operation of the computing device so as to execute the second command.

### Claim 3 (depends on 1)

The method of claim 1, wherein the thought-command mapping is remapped in response to detecting a change of the operation context during operation of the computing device.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
when an application currently executing on the computing device is a non-BCI-enabled application, dividing a screen area of the application into a plurality of cells each of which including a specific region of the screen area,
wherein each of the plurality of thought variants corresponds to a particular cell of the plurality of cells according to the thought-command mapping.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
detecting a first application currently executing on the computing device, the thought-command mapping being a first thought-command mapping corresponding to the first application.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
in response to a second application different from the first application being launched on the computing device, remapping the thought-command mapping such that the remapped thought-command mapping is a second thought-command mapping corresponding to the second application, the second thought-command mapping different from the first thought-command mapping.

### Claim 7 (depends on 5)

The method of claim 5, further comprising:
in response to a detection of a user action performed in the first application, remapping the thought-command mapping such that the remapped thought-command mapping is a second thought-command mapping based on the detected user action, the second thought-command mapping different from the first thought-command mapping.

### Claim 8 (depends on 5)

The method of claim 5, further comprising:
in response to a detection of an external event occurring during the execution of the first application, remapping the thought-command mapping such that the remapped thought-command mapping is a second thought-command mapping based on the detected external event, the second thought-command mapping different from the first thought-command mapping.

### Claim 9 (depends on 8)

The method of claim 8, wherein the external event is detected when the computing device receives an incoming electronic message.

### Claim 10 (depends on 8)

The method of claim 8, wherein the external event is detected when the computing device detects a change of a location of the user.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
in response to a detection of a user history associated with the first application, remapping the thought-command mapping such that the remapped thought-command mapping is a second thought-command mapping based on the detected user history, the second thought-command mapping different from the first thought-command mapping.

### Claim 12 (depends on 11)

The method of claim 11, wherein the user history is associated with at least one of contacts of the user, previously used words or phrases, and previously executed commands.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
controlling a display unit of the computing device so as to simultaneously display an application currently executing on the computing device and an image characterizing the thought-command mapping,
wherein the application currently executing on the computing device is displayed in a first screen area of the display unit, and the image characterizing the thought-command mapping is displayed in a second screen area of the display unit.

### Claim 14 (depends on 13)

The method of claim 13, wherein the image characterizing the thought-command mapping indicates each of the plurality of thought variants and each command executable by the computing device to which each of the plurality of thought variants corresponds.

### Claim 15 (depends on 13)

The method of claim 13, wherein the image characterizing the thought-command mapping includes a grid showing a correspondence between the plurality of thought variants and each command executable by the computing device.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
detecting a first application currently executing on the computing device, the thought-command mapping being a first thought-command mapping corresponding to the first application;
controlling a display unit of the computing device so as to display an image characterizing the first thought-command mapping;
in response to a second application different from the first application being launched on the computing device, remapping the thought-command mapping such that the remapped thought-command mapping is a second thought-command mapping corresponding to the second application, the second thought-command mapping different from the first thought-command mapping; and
controlling the display unit so as to update the displayed image such that the displayed image characterizes the second thought-command mapping.

### Claim 17 (depends on 1)

The method of claim 1, wherein the thought-command mapping includes a multi-length mapping scheme in which a first thought variant among the plurality of thought variants followed by one or more subsequent thought variants among the plurality of thought variants corresponds to a command executable by the computing device.

### Claim 18 (depends on 1)

The method of claim 1, wherein the plurality of thought variants include thoughts of a particular body part, thoughts of a particular sound or word, thoughts of a particular image, thoughts of a particular object visible to the user, motor imagery, or any combination thereof.

### Claim 19 (depends on 1)

The method of claim 1, further comprising:
controlling operation of an external device in communication with the computing device through the BCI based on the at least one measured brain signal according to the thought-command mapping.

### Claim 20

A system comprising:
one or more brain sensors configured to be coupled to a brain of a user and to measure at least one brain signal of the user; and
a computing device including a memory configured to store at least one program instruction and a processor configured to execute the at least one program instruction, the computing device in operative communication with the brain of the user through a brain-computer interface (BCI),
wherein the processor is configured to:
define a thought-command mapping according to which operation of the computing device is configured to be controlled through the BCI based on the at least one measured brain signal, the thought-command mapping defining a plurality of thought variants, a first thought variant among the plurality of thought variants corresponding to a first command executable by the computing device, each of the plurality of thought variants configured to be sensed based on a signal analysis of the at least one measured brain signal,
detect an operation context of the computing device during operation of the computing device, and
remap the thought-command mapping based on the detected operation context during operation of the computing device such that the first thought variant corresponds to a second command executable by the computing device different from the first command, and
wherein, in response to the remapping of the thought-command mapping, the operation of the computing device is configured to be controlled through the BCI based on the at least one measured brain signal according to the remapped thought-command mapping, and
wherein the operation context is associated with an application currently executing on the computing device, an external event occurring during operation of the computing device, and a user history associated with the computing device.

### Claim 21

A method comprising:
activating a muscle-computer interface (muCI) through which a computing device is in operative communication with one or more muscles of a user;
defining a muscle activity-command mapping according to which operation of the computing device is configured to be controlled through the muCI based on at least one muscle signal of the user measured via one or more muscle sensors coupled to the one or more muscles of the user, the muscle activity-command mapping defining a plurality of muscle activity variants each of which corresponding a command executable by the computing device, a first muscle activity variant among the plurality of muscle activity variants corresponding to a first command executable by the computing device;
detecting an operation context of the computing device during operation of the computing device; and
remapping the muscle activity-command mapping based on the detected operation context during operation of the computing device such that the first muscle activity variant corresponds to a second command executable by the computing device different from the first command,
wherein, in response to the remapping of the muscle activity-command mapping, the operation of the computing device is configured to be controlled through the muCI based on the at least one measured muscle signal according to the remapped muscle activity-command mapping, and
wherein the operation context is associated with an application currently executing on the computing device, an external event occurring during operation of the computing device, and a user history associated with the computing device.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
measuring the at least one muscle signal of the user using the one or more muscle sensors;
sensing the first muscle activity variant based on a signal analysis of the at least one measured muscle signal;
determining that the second command corresponds to the first muscle activity variant according to the remapped muscle activity-command mapping; and
controlling the operation of the computing device so as to execute the second command.

### Claim 23 (depends on 21)

The method of claim 21, wherein the muscle activity-command mapping is remapped in response to detecting a change of the operation context during operation of the computing device.

### Claim 24 (depends on 21)

The method of claim 21, further comprising:
when an application currently executing on the computing device is a non-muCI-enabled application, dividing a screen area of the application into a plurality of cells each of which including a specific region of the screen area,
wherein each of the plurality of muscle activity variants corresponds to a particular cell of the plurality of cells according to the muscle activity-command mapping.

### Claim 25 (depends on 21)

The method of claim 21, further comprising:
detecting a first application currently executing on the computing device, the muscle activity-command mapping being a first muscle activity-command mapping corresponding to the first application; and
in response to a second application different from the first application being launched on the computing device, remapping the thought-command mapping such that the remapped thought-command mapping is a second thought-command mapping corresponding to the second application, the second thought-command mapping different from the first thought-command mapping.

### Claim 26 (depends on 21)

The method of claim 21, further comprising:
controlling a display unit of the computing device so as to simultaneously display an application currently executing on the computing device and an image characterizing the muscle activity-command mapping,
wherein the application currently executing on the computing device is displayed in a first screen area of the display unit, and the image characterizing the muscle activity-command mapping is displayed in a second screen area of the display unit.

### Claim 27 (depends on 21)

The method of claim 21, further comprising:
detecting a first application currently executing on the computing device, the muscle activity-command mapping being a first muscle activity-command mapping corresponding to the first application;
controlling a display unit of the computing device so as to display an image characterizing the first muscle activity-command mapping;
in response to a second application different from the first application being launched on the computing device, remapping the muscle activity-command mapping such that the remapped muscle activity-command mapping is a second muscle activity-command mapping corresponding to the second application, the second muscle activity-command mapping different from the first muscle activity-command mapping; and
controlling the display unit so as to update the displayed image such that the displayed image characterizes the second muscle activity-command mapping.

### Claim 28

A system comprising:
one or more muscle sensors configured to be coupled to one or more muscles of a user and to measure at least one muscle signal of the user; and
a computing device including a memory configured to store at least one program instruction and a processor configured to execute the at least one program instruction, the computing device in operative communication with the one or more muscles of the user through a muscle-computer interface (muCI),
wherein the processor is configured to:
define a muscle activity-command mapping according to which operation of the computing device is configured to be controlled through the muCI based on the at least one measured muscle signal, the muscle activity-command mapping defining a plurality of muscle activity variants, a first muscle activity variant among the plurality of muscle activity variants corresponding to a first command executable by the computing device, each of the plurality of muscle activity variants configured to be sensed based on a signal analysis of the at least one measured muscle signal,
detect an operation context of the computing device during operation of the computing device, and
remap the muscle activity-command mapping based on the detected operation context during operation of the computing device such that the first muscle activity variant corresponds to a second command executable by the computing device different from the first command, and
wherein, in response to the remapping of the muscle activity-command mapping, the operation of the computing device is configured to be controlled through the muCI based on the at least one measured muscle signal according to the remapped muscle activity-command mapping, and
wherein the operation context is associated with an application currently executing on the computing device, an external event occurring during operation of the computing device, and a user history associated with the computing device.