# US 2018/0024635 A1 · Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0024635 A1
- Document ID: US20180024635A1
- Status: Published application
- Kind: A1
- Application number: 15/659,072
- Filed: 2017-07-25
- Published: 2018-01-25
- Earliest claimed: 2016-07-25
- Assignees: Facebook Technologies, LLC
- Inventors: Patrick Kaifosh; Timothy Machado; Thomas Reardon; Erik Schomburg
- Web page: https://patentgrep.com/patent/US20180024635A1

## CPC

- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/02: Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and apparatus for providing a dynamically-updated computerized musculo-skeletal representation comprising a plurality of rigid body segments connected by joints. The method comprises recording, using a plurality of autonomous sensors arranged on one or more wearable devices, a plurality of autonomous signals from a user, wherein the plurality of autonomous sensors include a plurality of neuromuscular sensors configured to record neuromuscular signals. The method further comprises providing as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals. The method further comprises determining, based on an output of the trained statistical model, musculo-skeletal position information describing a spatial relationship between two or more connected segments of the plurality of rigid body segments of the computerized musculo-skeletal representation, and updating the computerized musculo-skeletal representation based, at least in part, on the musculo-skeletal position information.

## Claims

### Claim 1

A computerized system for providing a dynamically-updated computerized musculo-skeletal representation comprising a plurality of rigid body segments connected by joints, the system comprising:
a plurality of autonomous sensors including a plurality of neuromuscular sensors, wherein the plurality of autonomous sensors are arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors are configured to continuously record a plurality of neuromuscular signals from a user; and
at least one computer processor programmed to:
provide as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromusuclar signals;
determine, based on an output of the trained statistical model, musculo-skeletal position information describing a spatial relationship between two or more connected segments of the plurality of rigid body segments of the computerized musculo-skeletal representation; and
update the computerized musculo-skeletal representation based, at least in part, on the musculo-skeletal position information.

### Claim 2 (depends on 1)

The computerized system of claim 1, wherein the determined musculo-skeletal position information comprises a set of joint angles describing the spatial relationship between the two or more connected segments of the computerized musculo-skeletal representation

### Claim 3 (depends on 1)

The computerized system of claim 1, wherein the plurality of autonomous sensors further includes at least one IMU sensor configured to continuously record IMU signals, wherein the at least one processor is further programmed to provide as input to the trained statistical model, the plurality of IMU signals and/or information based on the plurality of IMU signals, and wherein determining, based on an output of the trained statistical model, musculo-skeletal position information comprises determining musculo-skeletal position information based, at least in part, on the neuromuscular signals and the IMU signals provided as input to the trained statistical model.

### Claim 4 (depends on 3)

The computerized system of claim 3, wherein the at least one IMU sensor is configured to continuously record the IMU signals at a first sampling rate, the plurality of neuromuscular sensors are configured to continuously record the neuromuscular signals at a second sampling rate, the first sampling rate and the second sampling rate being different, and
wherein
(1) at least one of the IMU signals or the neuromuscular signals are resampled such that the IMU signals and the neuromuscular signals are provided as input to the trained statistical model at the same rate; or
(2) wherein the trained statistical model is configured to process asynchronous inputs.

### Claim 5 (depends on 4)

The computerized system of claim 4, wherein the at least one IMU sensor and the plurality of neuromuscular sensors are arranged on a same wearable device.

### Claim 6 (depends on 1)

The computerized system of claim 1, wherein the neuromuscular sensors comprise electromyography (EMG) sensors, mechanomyography (MMG) sensors, sonomyography (SMG) sensors, or a combination of EMG, MMG and SMG sensors.

### Claim 7 (depends on 1)

The computerized system of claim 1, wherein the at least one computer processor is further programmed to provide a modified musculo-skeletal representation by combining information determined based on the neuromuscular signals recorded by the neuromuscular sensors with the generated musculo-skeletal representation.

### Claim 8 (depends on 8)

The computerized system of claim 8, wherein the information determined based on the neuromuscular signals comprises force information.

### Claim 9 (depends on 1)

The computerized system of claim 1, wherein the at least one processor is further programmed to send based, at least in part, on the updated computerized musculo-skeletal representation, one or more control signals to a controller configured to instruct a device to perform an action based on the one or more control signals.

### Claim 10 (depends on 9)

The computerized system of claim 9, wherein the at least one processor is further programmed to execute a computer application that provides a virtual reality environment,
wherein the controller comprises a display controller configured to instruct a display to display a visual representation of a character in the virtual reality environment, and
wherein the one or more control signals comprise signals to instruct the display controller to update in real time the visual representation of the character based on the updated computerized musculo-skeletal representation.

### Claim 11 (depends on 10)

The computerized system of claim 10, wherein the virtual reality environment comprises a virtual object and wherein updating the visual representation of the character based on the updated computerized musculo-skeletal representation comprises updating the visual representation such that the character interacts with the virtual object.

### Claim 12 (depends on 11)

The computerized system of claim 11, wherein interacting with the virtual object comprises an action selected from the group consisting of grasping the virtual object, dropping the virtual object, pushing the virtual object, throwing the virtual object, pulling the virtual object, and opening the virtual object, and closing the virtual object.

### Claim 13 (depends on 9)

The computerized system of claim 9, wherein the controller includes a control interface for a physical device, and wherein the one or more control signals comprise signals to instruct at least one component of the physical device to move based on the updated computerized musculo-skeletal representation.

### Claim 14 (depends on 1)

The computerized system of claim 1, wherein the trained statistical model comprises a recurrent neural network.

### Claim 15 (depends on 1)

The computerized system of claim 1, wherein the trained statistical model comprises a non-linear regression model.

### Claim 16 (depends on 1)

The computerized system of claim 1, wherein the one or more wearable devices consists of a single wearable device, and wherein all of the plurality of autonomous sensors are arranged on the single wearable device configured to be worn on or around a body part of the user.

### Claim 17 (depends on 1)

The computerized system of claim 1, wherein at least one of the one or more wearable devices comprises a flexible or elastic band configured to be worn around the body part of the user.

### Claim 18 (depends on 16)

The computerized system of claim 16, wherein the flexible or elastic band comprises an armband configured to be worn around an arm of the user.

### Claim 19

A method of providing a dynamically-updated computerized musculo-skeletal representation comprising a plurality of rigid body segments connected by joints, the method comprising:
recording, using a plurality of autonomous sensors arranged on one or more wearable devices, a plurality of autonomous signals from a user, wherein the plurality of autonomous sensors comprise a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals;
providing as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals;
determining, based on an output of the trained statistical model, musculo-skeletal position information describing a spatial relationship between two or more connected segments of the plurality of rigid body segments of the computerized musculo-skeletal representation; and
updating the computerized musculo-skeletal representation based, at least in part, on the musculo-skeletal position information.

### Claim 20

A computer system for training a statistical model to predict musculo-skeletal position information based, at least in part, on autonomous signals recorded by a plurality of autonomous sensors, wherein the plurality of autonomous sensors include a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals, the computer system comprising:
an input interface configured to:
receive the neuromuscular signals recorded during performance of a task performed by one or more users;
receive position information indicating a position of a plurality of rigid body segments of a computerized musculo-skeletal representation during performance of the task performed by the one or more users; and
at least one storage device configured to store a plurality of instructions that, when executed by at least one computer processor perform a method of:
generating training data based, at least on part, on the received neuromuscular signals and the received position information;
training the statistical model using at least some of the generated training data to output a trained statistical model; and
storing, by the at least one storage device, the trained statistical model, wherein the trained statistical model is configured to predict musculo-skeletal position information based, at least in part on continuously recorded signals from the neuromuscular sensors.
