# US 2018/0020951 A1 · Adaptive system for deriving control signals from measurements of neuromuscular activity

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0020951 A1
- Document ID: US20180020951A1
- Status: Published application
- Kind: A1
- Application number: 15/659,487
- Filed: 2017-07-25
- Published: 2018-01-25
- Earliest claimed: 2016-07-25
- Assignees: Facebook Technologies, LLC
- Inventors: Patrick Kaifosh; Timothy Machado; Thomas Reardon; Erik Schomburg; Joshua Merel; Steven Demers
- Web page: https://patentgrep.com/patent/US20180020951A1

## CPC

- A61B 5/1106: {to assess neuromuscular blockade, e.g. to estimate depth of anaesthesia}
- A61B 5/313: for electromyography [EMG]
- A61B 5/378: Visual stimuli
- A61B 5/395: Details of stimulation, e.g. nerve stimulation to elicit EMG response
- A61B 5/681: {Wristwatch-type devices}
- A61F 2/72: Bioelectric control, e.g. myoelectric
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- A61B 5/296: for electromyography [EMG]
- A61B 5/38: Acoustic or auditory stimuli
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and apparatus for adapting a control mapping associating sensor signals with control signals for controlling an operation of a device. The method comprises obtaining first state information for an operation of the device, providing the first state information as input to an intention model associated with an operation of the device and obtaining corresponding first intention model output, providing a plurality of neuromuscular signals recorded from a user and/or signals derived from the neuromuscular signals as inputs to a first control mapping and obtaining corresponding first control mapping output, and updating the first control mapping using the inputs provided to the first control mapping and the first intention model output to obtain a second control mapping.

## Claims

### Claim 1

A system for adapting a control mapping associating sensor signals with control signals for controlling an operation of a device, the system comprising:
a plurality of sensors including a plurality of neuromuscular sensors arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors are configured to continuously record neuromuscular signals from a user;
at least one computer hardware processor;
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
obtaining first state information for an operation of the device;
providing the first state information as input to an intention model associated with the operation of the device and obtaining corresponding first intention model output;
providing the neuromuscular signals and/or signals derived from the neuromuscular signals as inputs to a first control mapping and obtaining corresponding first control mapping output; and
updating the first control mapping using the inputs to the first control mapping and the first intention model output to obtain a second control mapping.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
obtaining a new set of neuromuscular signals from the plurality of neuromuscular sensors; and
providing the new set of neuromuscular signals and/or signals derived from the new set of neuromuscular signals as inputs to the second control mapping to obtain corresponding second control mapping output; and
controlling the device using the second control mapping output.

### Claim 3 (depends on 2)

The system of claim 2, wherein the processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
obtaining second state information for the at least one user-interactive application; providing the second state information as input to the intention model associated with the operation of the device and obtaining corresponding second intention model output; and
updating the second control mapping using the inputs provided to the second control mapping and the second intention model output to obtain a third control mapping.

### Claim 4 (depends on 1)

The system of claim 1, wherein updating the first control mapping using the inputs provided to the first control mapping and the first intention model output to obtain a second control mapping is performed using a recursive least squares (RLS) technique.

### Claim 5 (depends on 4)

The system of claim 4, wherein the RLS technique is configured to fit a linear model that predicts velocity control signals indicating the user's control intentions from the neuromuscular signals.

### Claim 6 (depends on 1)

The system of claim 1, wherein the intention model is a trained statistical model implemented as a recurrent neural network.

### Claim 7 (depends on 6)

The system of claim 6, wherein the recurrent neural network is a long short-term memory (LSTM) neural network.

### Claim 8 (depends on 1)

The system of claim 1, wherein the intention model is trained statistical model implemented as a non-linear regression model.

### Claim 9 (depends on 1)

The system of claim 1, wherein the neuromuscular sensors comprise electromyography (EMG) sensors, mechanomyography (MMG) sensors, sonomyography (SMG) sensors, or a combination of EMG, MMG and SMG sensors.

### Claim 10 (depends on 1)

The system of claim 1, wherein the one or more wearable devices consists of a single wearable device, and wherein all of the plurality of sensors are arranged on the single wearable device configured to be worn on or around a body part of the user.

### Claim 11 (depends on 1)

The system of claim 1, wherein at least one of the one or more wearable devices comprises a flexible or elastic band configured to be worn around the body part of the user.

### Claim 12 (depends on 11)

The system of claim 11, wherein the flexible or elastic band comprises an armband configured to be worn around an arm of the user.

### Claim 13 (depends on 1)

The system of claim 1, wherein the plurality of sensors further includes at least one IMU sensor configured to continuously record IMU signals, wherein the processor-executable instructions, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
providing the IMU signals and/or information based on the plurality of IMU signals as inputs to the first control mapping.

### Claim 14 (depends on 13)

The system of claim 13, wherein the at least one IMU sensor and the plurality of neuromuscular sensors are arranged on a same wearable device.

### Claim 15 (depends on 1)

The system of claim 1, wherein updating the first control mapping further comprises updating the first control mapping using the first control mapping output.

### Claim 16 (depends on 1)

The system of claim 1, wherein the intention model is generated based on heuristic or intuitive approximation of user interactions with the device or observed user interactions by one or more users with the device.

### Claim 17 (depends on 1)

The system of claim 1, wherein the device comprises a computer executing a user-interactive application, and wherein the operation of the device comprises an operation of the user-interactive application.

### Claim 18 (depends on 1)

The system of claim 1, wherein the processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
controlling the device using the first control mapping output.

### Claim 19

A method of adapting a control mapping associating sensor signals with control signals for controlling an operation of a device, the method comprises;
obtaining a plurality of sensor signals from a plurality of sensors including a plurality of neuromuscular sensors arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors are configured to continuously record neuromuscular signals from a user;
obtaining first state information for an operation of a device;
providing the first state information as input to an intention model associated with the operation of the device and obtaining corresponding first intention model output;
providing the neuromuscular signals and/or signals derived from the neuromuscular signals as inputs to a first control mapping and obtaining corresponding first control mapping output; and
updating the first control mapping using the inputs provided to the first control mapping and the first intention model output to obtain a second control mapping.

### Claim 20

A non-transitory computer-readable storage medium encoded with a plurality of instructions that, when executed by at least one computer hardware processor perform a method comprising:
obtaining a plurality of sensor signals from a plurality of sensors including a plurality of neuromuscular sensors arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors are configured to continuously record neuromuscular signals from a user;
obtaining first state information for an operation of a device;
providing the first state information as input to an intention model associated with the operation of the device and obtaining corresponding first intention model output;
providing the neuromuscular signals and/or signals derived from the neuromuscular signals as inputs to a first control mapping and obtaining corresponding first control mapping output; and
updating the first control mapping using the inputs provided to the first control mapping and the first intention model output to obtain a second control mapping.
