# US 2020/0401224 A1 · Wearable joint tracking device with muscle activity and methods thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401224 A1
- Document ID: US20200401224A1
- Status: Published application
- Kind: A1
- Application number: 16/908,337
- Filed: 2020-06-22
- Published: 2020-12-24
- Earliest claimed: 2019-06-21
- Assignees: Rehabilitation Institute of Chicago d/b/a Shirley Ryan AbilityLab
- Applicants: Rehabilitation Institute of Chicago d/b/a Shirley Ryan AbilityLab
- Inventors: Ronald Cotton
- Web page: https://patentgrep.com/patent/US20200401224A1

## CPC

- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/1122: {of movement trajectories}
- A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)}
- A61B 5/1495: Calibrating or testing of in-vivo probes
- A61B 5/224: {Measuring muscular strength}
- A61B 5/313: for electromyography [EMG]
- A61B 5/4528: {Joints (A61B5/4533, A61B5/4538 take precedence)}
- A61B 5/458: {Evaluating the elbow}
- A61B 5/4585: {Evaluating the knee}
- A61B 5/486: {Biofeedback (using electroencephalography [EEG] A61B5/375)}
- A61B 5/6802: {Sensor mounted on worn items}
- A61B 5/742: {using visual displays (displays for heart-related electrical signals, e.g. ECG, A61B5/339)}
- A61B 5/1123: {Discriminating type of movement, e.g. walking or running (A61B5/1116, A61B5/112 take precedence)}
- A61B 5/257: using adhesive means, e.g. adhesive pads or tapes
- A61B 5/7225: {Details of analogue processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation (input circuits for detecting, measuring, or recording bioelectric or biomagnetic signals A61B5/30; specific diagnostic methods using bioelectric or biomagnetic signals A61B5/316)}
- A61B 2505/09: Rehabilitation or training
- A61B 2560/0214: of power generation or supply
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61B 2562/164: the sensor is mounted in or on a conformable substrate or carrier
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

A system for utilizing wearable, wireless-connected sensors that record both IMU and EMG muscle activity for rehabilitation with kinematic monitoring is disclosed. The system runs an efficient quaternion-based complementary filter that estimates the sensor orientation while correcting for estimate drift and constraining magnetometer estimates to only influence heading. The difference in the two sensor orientations is used to estimate the joint angle, which can be further improved with joint axis estimation. Thus, successful tracking of joint angle and muscle activity in a home environment can be accomplished with the sensors and a smartphone. Further, a system utilizing these sensors may include video-capable cameras configured for capturing a video of a pose of the user, and a game framework executable by a processor in communication with the sensors and the cameras. The system may include a game framework configured to utilize a pose trajectory and one or more joint angles determined using the sensors to provide biofeedback to a user.

## Claims

### Claim 1

A system for utilizing wearable sensors to record muscle activity and provide biofeedback for rehabilitation with kinematic monitoring, comprising:
a plurality of sensors aligned on a limb of a user, each of the plurality of sensors including an inertial measurement unit (IMU) for generating inertial data and a plurality of electromyography (EMG) electrodes for generating muscle EMG data; and
a processor in operable communication with the plurality of sensors, the processor configured to:
estimate a pose of the limb of the user, by estimating an attitude of a first sensor and an attitude of a second sensor of the plurality of sensors,
estimate a joint angle of the limb, the joint angle measured as a rotation magnitude about an inferred axis of the limb defined by the attitude of the first sensor and the attitude of the second sensor, and
execute a therapy application, the therapy application configured to utilize the EMG data, and pose and joint angle of the limb as estimated from the plurality of sensors to detect muscle activity and movement related to the limb and provide biofeedback to a user for rehabilitation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor, by executing the therapy application, is configured to cue, via a display, an instruction to apply a predetermined movement of the limb or muscle activity of the limb for a predetermined therapy protocol, and provide visual biofeedback related to a response of the user to the instruction as expressed by the muscle activity and movement related to the limb as detected.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processor, by executing the therapy application, provides a game to the user with visual interaction associated with the game provided to the user via a display, and the processor tracks changes in the muscle activity and movement related to the limb over a predetermined time period as the user engages the game to pursue a target activation level associated with the game related to muscle control or muscle strength.

### Claim 4 (depends on 3)

The system of claim 3, wherein processor, by executing the game as configured, alters visual feedback presented to the user via the display according to an upper bound and a lower bound, and range around the target activation level associated with a given therapy task to provide visual biofeedback therapy.

### Claim 5 (depends on 1)

The system of claim 1, wherein each of the plurality of sensors implement firmware that digitizes the muscle EMG data, and applies additional digital bandpass filtering to the muscle EMG data, the muscle EMG data communicated to the processor in parallel with the inertial data to accommodate simultaneous monitoring to changes in the inertial data and the muscle EMG data.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor measures the orientation of a sensor of the plurality of sensors by transforming magnetomer, accelerometer, and gyroscope measurements of the inertial data.

### Claim 7 (depends on 6)

The system of claim 6, wherein the processor combines rotation angles from multiple sensors of the plurality of sensors to estimate a plurality of joint angles.

### Claim 8

A system, comprising:
a plurality of sensors aligned on a body part of a user, each of the plurality of sensors including an inertial measurement unit (IMU) for generating inertial data and a plurality of electromyography (EMG) electrodes for generating muscle EMG data;
a plurality of video-capable cameras configured for capturing a video of a pose of the user, the video including a plurality of frames; and
a processor in operable communication with the plurality of sensors and the plurality of video-capable cameras, the processor including instructions that, when executed, cause the processor to:
collect the video from the plurality of video-capable cameras and the inertial data from each of the plurality of sensors simultaneously;
extract joint locations for each joint in each frame of the plurality of frames of the video;
determine an initial pose estimate for each frame of the plurality of frames of the video;
optimize a pose trajectory to fit the joint locations based on the initial pose estimate from each frame of the video and the inertial data from each of the plurality of sensors to determine biomechanical constraints;
determine the calibration of sensor orientations relative to body segment orientations; and
determine one or more joint angles for the body parts of the user.

### Claim 9 (depends on 8)

The system of claim 8, wherein the initial pose estimate is determined and the pose trajectory is modeled using a skinned multi-person linear model.

### Claim 10 (depends on 8)

The system of claim 8, wherein the steps of extracting joint locations and determining an initial pose estimate for each frame of the plurality of frames produce a sequence of joint keypoints with associated confidences for each timestamp, joint, and camera view of the video.

### Claim 11 (depends on 10)

The system of claim 10, wherein the step of optimizing the pose trajectory includes determining a predicted location of each of the joint keypoints by application of one or more smoothing factors and the biomechanical constraints.

### Claim 12 (depends on 8)

The system of claim 8, wherein the processor is operable to execute instructions that, when executed, further cause the processor to:
calibrate an orientation of each sensor of the plurality of sensors relative to the body part.

### Claim 13 (depends on 8)

The system of claim 8, wherein joint angles are inferred using the biomechanical constraints.

### Claim 14 (depends on 8)

The system of claim 8, wherein joint angles are determined using a relative rotation of each of the plurality of sensors.

### Claim 15 (depends on 8)

The system of claim 8, further comprising:
a game framework executable by the processor, the game framework configured to utilize the pose trajectory and one or more joint angles determined using the one or more sensors to provide biofeedback to a user for rehabilitation.