# US 2023/0274145 A1 · Method and system for symmetric recognition of handed activities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274145 A1
- Document ID: US20230274145A1
- Status: Published application
- Kind: A1
- Application number: 18/311,825
- Filed: 2023-05-03
- Published: 2023-08-31
- Earliest claimed: 2019-03-15
- Assignees: Hinge Health, Inc.
- Applicants: Hinge Health, Inc.
- Inventors: Colin Brown; Andrey Tolstikhin
- Web page: https://patentgrep.com/patent/US20230274145A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/82: using neural networks
- G06V 40/107: {Static hand or arm}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06N 3/048: Activation functions
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

This disclosure describes an activity recognition system for asymmetric (e.g., left- and right- handed) activities that leverages the symmetry intrinsic to most human and animal bodies. Specifically, described is 1) a human activity recognition system that only recognizes handed activities but is inferenced twice, once with input flipped, to identify both left- and right- handed activities and 2) a training method for learning-based implementations of the aforementioned system that flips all training instances (and associated labels) to appear left-handed and in doing so, balances the training dataset between left- and right-handed activities.

## Claims

### Claim 1

A method of training an activity classifier, the method comprising:
identifying training instances associated with a first handedness, wherein each training instance is associated with a corresponding class label and a corresponding segment of activity data;
for each training instance,
modifying that training instance by -
transforming the corresponding class label to a class label of a second handedness opposite the first handedness, and
transforming the corresponding segment of activity data using a transformation that is associated with the first handedness; and
training the activity classifier using the modified training instances,
wherein as a result of said modifying, none of the modified training instances are labeled with the first handedness.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the first handedness is left handedness and the second handedness is right handedness, and
wherein said modifying causes (i) left-handed labels to be replaced with right-handed labels and (ii) the activity data to be horizontally flipped.

### Claim 3 (depends on 1)

The method of claim 1,
wherein the first handedness is right handedness and the second handedness is left handedness, and
wherein said modifying causes (i) right-handed labels to be replaced with left-handed labels and (ii) the activity data to be horizontally flipped.

### Claim 4 (depends on 1)

The method of claim 1, wherein each corresponding segment of activity data includes a set of keypoints representing skeletal joints of a corresponding person as an array of X and Y coordinates.

### Claim 5 (depends on 4)

The method of claim 4, wherein the transformation is a horizontal transformation that causes each X coordinate to be reversed while leaving each Y coordinate unaltered.

### Claim 6 (depends on 1)

The method of claim 1, wherein each corresponding segment of activity data includes keypoints representing skeletal joints of a corresponding person, images of the corresponding person, or video of the corresponding person.

### Claim 7 (depends on 1)

The method of claim 1, wherein the activity classifier is implemented as a long-short-term memory (LSTM) recurrent neural network that comprises a series of fully connected layers, activation layers, LSTM layers, and softmax layers.

### Claim 8 (depends on 7)

The method of claim 7, wherein at least some of the fully connected layers and the LSTM layers are parameterized by weights that are learned by training on the modified training instances.

### Claim 9

A method of training a machine learning model, the method comprising:
identifying training instances associated with a first asymmetry, wherein each training instance is associated with a corresponding class label and a corresponding segment of activity data;
for each training instance,
modifying that training instance by -
transforming the corresponding class label to a class label of a second asymmetry opposite the first asymmetry, and
transforming the corresponding segment of activity data; and
training the machine learning model using the modified training instances,
wherein as a result of said modifying, none of the modified training instances are labeled with the first asymmetry.

### Claim 10 (depends on 9)

The method of claim 9,
wherein each corresponding segment of activity data is transformed by applying a transformation thereto, and
wherein the transformation is associated with the first asymmetry.

### Claim 11 (depends on 10)

The method of claim 10, wherein each corresponding segment of activity data includes a set of keypoints representing skeletal joints of a corresponding person as an array of X and Y coordinates.

### Claim 12 (depends on 11)

The method of claim 11, wherein the transformation is a horizontal transformation that causes each X coordinate to be reversed while leaving each Y coordinate unaltered.

### Claim 13 (depends on 11)

The method of claim 11, wherein the transformation is a vertical transformation that causes each Y coordinate to be reversed while leaving each X coordinate unaltered.

### Claim 14 (depends on 9)

The method of claim 9,
wherein the first asymmetry is left handedness,
wherein said modifying causes each training instance to be associated with a right-handed label, and
wherein said training causes the machine learning model to be trained to classify right-handed activities represented in the training instances as left-handed activities.

### Claim 15 (depends on 9)

The method of claim 9,
wherein the first asymmetry is right handedness,
wherein said modifying causes each training instance to be associated with a left-handed label, and
wherein said training causes the machine learning model to be trained to classify left-handed activities represented in the training instances as right-handed activities.

### Claim 16 (depends on 9)

The method of claim 9, wherein the machine learning model is based on a neural network, a decision tree, a support vector machine, or a classifier.

### Claim 17 (depends on 9)

The method of claim 9, wherein the machine learning model is designed to accept, as input, activity data having a same format as the corresponding transformed segments of activity data associated with the modified training instances.

### Claim 18

A non-transitory medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:
identifying training instances associated with a first asymmetry, wherein each training instance is associated with a corresponding class label and a corresponding segment of activity data;
for each training instance,
modifying that training instance by -
transforming the corresponding class label to a class label of a second asymmetry opposite the first asymmetry, and
transforming the corresponding segment of activity data; and
training a machine learning model such that one or more weights are parameterized based on the modified training instances and the corresponding transformed segments of activity data.

### Claim 19 (depends on 18)

The non-transitory medium of claim 18, wherein the machine learning model is implemented as a long-short-term memory (LSTM) neural network that comprises inner product layers, activation layers, LSTM layers, and softmax layers.

### Claim 20 (depends on 19)

The non-transitory medium of claim 19, wherein the inner product layers and the LSTM layers are parameterized by the one or more weights.
