# US 2021/0232810 A1 · Automated monitoring system for biomechanical postural assessment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232810 A1
- Document ID: US20210232810A1
- Status: Published application
- Kind: A1
- Application number: 17/155,988
- Filed: 2021-01-22
- Published: 2021-07-29
- Earliest claimed: 2020-01-24
- Assignees: University of Washington
- Applicants: University of Washington
- Inventors: Behnoosh Parsa; Ashis G. Banerjee
- Web page: https://patentgrep.com/patent/US20210232810A1

## CPC

- G06T 7/73: using feature-based methods
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06N 3/048: Activation functions
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In some embodiments, a system is provided that comprises a computing system including at least one computing device and a camera communicatively coupled to the computing system via a network. The computing system is configured to generate a set of features based on three-dimensional joint location data representing postures of a subject over a plurality of time steps depicted in video data captured by the camera; provide the set of features to a first machine learning model trained to identify a start time step, an end time step, and an action identity for each action; provide the set of features to a second machine learning model trained to determine a postural assessment score for each time step; and determine an action score for each action based on the start time steps, the end time steps, the action identities, and the postural assessment scores for each time step.

## Claims

### Claim 1

A system, comprising:
a computing system including at least one computing device; and
a camera communicatively coupled to the computing system via a network;
wherein the computing system is configured to:
generate a set of features based on three-dimensional joint location data representing postures of a subject over a plurality of time steps depicted in video data captured by the camera;
provide the set of features to a first machine learning model trained to identify a start time step, an end time step, and an action identity for each action of the plurality of actions;
provide the set of features to a second machine learning model trained to determine a postural assessment score for each time step; and
determine an action score for each action of the plurality of actions based on the start time steps, the end time steps, the action identities, and the postural assessment scores for each time step.

### Claim 2 (depends on 1)

The system of claim 1, wherein the computing system is further configured to:
receive the video data from the camera; and
generate the three-dimensional joint location data based on the video data.

### Claim 3 (depends on 2)

The system of claim 2, wherein each time step of the plurality of time steps is represented by a frame of the video data.

### Claim 4 (depends on 3)

The system of claim 3, wherein the first machine learning model includes an encoder-decoder temporal convolutional network.

### Claim 5 (depends on 3)

The system of claim 3, wherein the second machine learning model includes a stacked long short-term memory model.

### Claim 6 (depends on 1)

The system of claim 1, wherein generating the set of features based on the three-dimensional joint location data includes providing the three-dimensional joint location data to a sequence of stacked graph convolutional networks.

### Claim 7 (depends on 1)

The system of claim 1, wherein the computing system is further configured to:
receive a set of training data, wherein each instance of training data in the set of training data is labeled with a plurality of training activities that each take place over a series of time steps, and wherein each training activity of the plurality of training activities is labeled with a postural assessment score.

### Claim 8 (depends on 7)

The system of claim 7, wherein the computing system is further configured to:
train the first machine learning model and the second machine learning model using the set of training data.

### Claim 9

A computer-implemented method of providing biomechanical analyses of movements of a subject, the method comprising:
generating, by a computing device, a set of features based on three-dimensional joint location data representing positions of the subject over a plurality of time steps while performing a plurality of actions;
providing, by the computing device, the set of features to a first machine learning model trained to identify a start time step, an end time step, and an action identity for each action of the plurality of actions;
providing, by the computing device, the set of features to a second machine learning model trained to determine a postural assessment score for each time step; and
determining, by the computing device, an action score for each action of the plurality of actions based on the start time steps, the end time steps, the action identities, and the postural assessment scores for each time step.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein providing the set of features to the first machine learning model includes providing the set of features to an encoder-decoder temporal convolutional network.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein providing the set of features to the second machine learning model includes providing the set of features and the action identities to a stacked long short-term memory (LSTM) model to generate the postural assessment scores.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, further comprising:
receiving, by the computing device, video data representing the movements of the subject while performing the plurality of actions; and
generating, by the computing device, the three-dimensional joint location data based on the video data.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein each time step of the plurality of time steps is represented by a frame of the video data.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein providing the set of features to the second machine learning model includes providing the set of features to a stacked long short-term memory (LSTM) model to obtain the postural assessment scores.

### Claim 15 (depends on 9)

The computer-implemented method of claim 9, wherein generating the set of features based on the three-dimensional joint location data includes providing the three-dimensional joint location data to a sequence of stacked graph convolutional networks.

### Claim 16

A computer-implemented method of training machine learning models to provide biomechanical analyses of movements of subjects, the method comprising:
receiving, by a computing device, a set of training data, wherein each instance of training data in the set of training data is labeled with a plurality of activities that each take place over a series of time steps, and wherein each activity of the plurality of activities is labeled with a postural assessment score;
generating, by the computing device, features for each instance of training data;
training, by the computing device, a first machine learning model to accept features of an instance of training data as input and to provide a plurality of action identities, start time steps, and end time steps as output;
training, by the computing device, a second machine learning model to accept features of an instance of training data as input and to provide a postural assessment score as output for each time step; and
storing, by the computing device, the first machine learning model and the second machine learning model in a model data store for processing new data.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein each instance of training data is a video of a subject, and wherein the method further comprises generating three-dimensional joint location data based on the video of the subject.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein generating the features for each instance of training data includes providing the three-dimensional joint location data to a sequence of stacked graph convolutional networks.

### Claim 19 (depends on 17)

The computer-implemented method of claim 17, wherein training the first machine learning model includes training an encoder-decoder temporal convolutional network.

### Claim 20 (depends on 17)

The computer-implemented method of claim 17, wherein training the second machine learning model includes training a stacked long short-term memory structure to provide postural assessment scores.
