# US 12,366,920 B2 · Systems and methods for gesture inference using transformations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,366,920 B2
- Document ID: US12366920B2
- Status: Granted
- Kind: B2
- Application number: 18/161,053
- Filed: 2023-01-28
- Published: 2024-03-28
- Granted: 2025-07-22
- Earliest claimed: 2022-09-26
- Assignees: Pison Technology, Inc.
- Applicants: Pison Technology, Inc.
- Inventors: Dexter Ang; David Cipoletta; Xiaofeng Tan; Matt Fleury; Dylan Pollack
- Web page: https://patentgrep.com/patent/US12366920B2

## CPC

- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/332: Portable devices specially adapted therefor
- A61B 5/681: {Wristwatch-type devices}
- G04G 21/025: {for measuring physiological data}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06N 5/04: Inference or reasoning models
- G06N 20/20: Ensemble learning
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G04G: ELECTRONIC TIME-PIECES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed are methods, systems and non-transitory computer readable memory for gesture inference. For instance, a first method may include computer vision to train and/or infer gesture inferences. For instance, a second method may include using transformations to data and/or ML models to address inter/intra-session variability of sensor data. For instance, a third method may include using ML model selection to select a ML model to address inter/intra-session variability of sensor data.

## Claims

### Claim 1

A system for gesture inference, the system comprising:
a wearable device configured to be worn on a portion of an arm of a user, the wearable device comprising:
a biopotential sensor, the biopotential sensor being configured to obtain biopotential data indicating electrical signals generated by nerves and muscles in the arm of the user; and
a motion sensor, the motion sensor being configured to obtain motion data relating to a motion of the portion of the arm of the user, the motion data and biopotential data collectively being sensor data; and
a processing pipeline configured to receive the biopotential data and the motion data and process the biopotential data and the motion data to generate a gesture inference output using a ML model, wherein the processing pipeline includes:
a pre-process module configured to:
obtain a first set of sensor data;
determine, based on the sensor data or a derivative thereof, a first transformation to the ML model and/or a second transformation to the first set of sensor data; and
apply the first transformation to the ML model to obtain a session ML model and/or apply the second transformation to the first set of sensor data or derivative thereof to obtain mapped sensor data; and
an inference module configured to infer the gesture inference based on (1) the session ML model and the first set of sensor data, and/or (2) the ML model and the mapped sensor data;
wherein the system is configured to, based on the gesture inference, determine a machine interpretable event, and execute an action corresponding to the machine interpretable event.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processing pipeline further includes an outlier detection module configured to confirm the first set of sensor data and/or derivatives of the first set of sensor data are not an outlier.

### Claim 3 (depends on 2)

The system of claim 2, wherein the outlier detection module compares the first set of sensor data and/or the derivatives of the first set of sensor data to at least one statistical file, a first statistical file of the at least one statistical file includes a set of values, the set of values indicating a multi-variable distribution based on historical sensor data for known gestures.

### Claim 4 (depends on 3)

The system of claim 3, wherein the outlier detection module uses the first statistical file during calibration and a second statistical file during inference.

### Claim 5 (depends on 4)

The system of claim 4, wherein the first statistical file is based on data gathered from a population of users, and the second statistical file is based on calibration data gathered from the user during calibration.

### Claim 6 (depends on 1)

The system of claim 1, wherein, to determine the first transformation to the ML model, the pre-process module is configured to:
determine, based on the sensor data or derivative thereof, that the wearable device is in a deviated state relative to the arm of the user, the deviated state being known to modify the sensor data received by the wearable device according to a known deviation pattern relative to the sensor data that would be received by the wearable device if the wearable device were in a neutral state; and
based on the determined deviated state, determine the first transformation to the ML model, the determined first transformation to the ML model being configured to improve inference accuracy of the ML model while the wearable device is in the deviated state.

### Claim 7 (depends on 6)

The system of claim 6, wherein the deviated state comprises a deviated arm posture that is different from a neutral arm posture when the wearable device is in a neutral state, and determining the first transformation to the ML model comprises:
determining, based on the motion data, that the arm of the user is in the deviated arm posture.

### Claim 8 (depends on 1)

The system of claim 1, wherein the first transformation applies adjustments to one or more model parameters, wherein the model parameters include one or more of weights, biases, thresholds, or values of the ML model.

### Claim 9 (depends on 1)

The system of claim 1, wherein, to determine the second transformation to the first set of sensor data, the pre-process module is configured to:
determine, based on the sensor data, that the wearable device is in a deviated state relative to the arm of the user, the deviated state being known to modify the sensor data received by the wearable device according to a known deviation pattern relative to the sensor data that would be received by the wearable device if the wearable device were in a neutral state; and
based on the determined deviated state, determine the second transformation to the first set of sensor data, the determined second transformation to the first set of sensor data being configured produce the mapped sensor data, which is more similar to the sensor data that would be received by the wearable device if the wearable device were in the neutral state than is the first set of sensor data.

### Claim 10 (depends on 1)

The system of claim 1, wherein the second transformation comprises a rotation, a translation, a projection, and/or a scaling, so that the first set of sensor data, as transformed to the mapped sensor data, is more similar to sensor data that would be received by the wearable device if the wearable device were in a neutral state.

### Claim 11 (depends on 1)

The system of claim 1, wherein the ML model includes a first ML model to infer an IMU gesture inference based on the motion data, and a second ML model to infer a biopotential gesture inference based on the biopotential data and the motion data, and the inference module determines the gesture inference based on the IMU gesture inference and the biopotential gesture inference.

### Claim 12 (depends on 11)

The system of claim 11, wherein the inference module is configured to:
store successive biopotential gesture inferences;
determine a threshold number of the successive biopotential gesture inferences have been stored; and
determine the gesture inference based on the threshold number of the successive biopotential gesture inferences and probability information of a confusion matrix, the confusion matrix having been generated during training of the second ML model.

### Claim 13

A computer-implemented method for gesture inference, the computer-implemented method comprising:
obtaining a first set of sensor data, the first set of sensor data including motion data and biopotential data, the motion data being obtained by a motion sensor of a wearable device, the motion sensor being configured to obtain the motion data relating to a motion of a portion of an arm of a user wearing the wearable device, the biopotential data being obtained by a biopotential sensor of the wearable device, the biopotential sensor being configured to obtain the biopotential data indicating electrical signals generated by nerves and muscles in the arm of the user;
determining, based on the sensor data or a derivative thereof, a first transformation to a ML model and/or a second transformation to the first set of sensor data;
applying the first transformation to the ML model to obtain a session ML model and/or applying the second transformation to the first set of sensor data or derivative thereof to obtain mapped sensor data; and
inferring a gesture inference based on (1) the session ML model and the first set of sensor data, and/or (2) the ML model and the mapped sensor data;
wherein the wearable device is configured to, based on the gesture inference, determine a machine interpretable event, and execute an action corresponding to the machine interpretable event.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein, to determine the first transformation to the ML model, the computer-implemented method further includes:
determining, based on the sensor data or derivative thereof, that the wearable device is in a deviated state relative to the arm of the user, the deviated state being known to modify the sensor data received by the wearable device according to a known deviation pattern relative to the sensor data that would be received by the wearable device if the wearable device were in a neutral state; and
based on the determined deviated state, determining the first transformation to the ML model, the determined first transformation to the ML model being configured to improve inference accuracy of the ML model while the wearable device is in the deviated state.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein the deviated state comprises a deviated arm posture that is different from a neutral arm posture when the wearable device is in a neutral state, and, to determine the first transformation to the ML model, the computer-implemented method further comprises:
determining, based on the motion data, that the arm of the user is in the deviated arm posture.

### Claim 16 (depends on 13)

The computer-implemented method of claim 13, wherein the first transformation applies adjustments to one or more model parameters, wherein the model parameters include one or more of weights, biases, thresholds, or values of the ML model.

### Claim 17 (depends on 13)

The computer-implemented method of claim 13, wherein, to determine the second transformation to the first set of sensor data, the computer-implemented method further includes:
determining, based on the sensor data, that the wearable device is in a deviated state relative to the arm of the user, the deviated state being known to modify the sensor data received by the wearable device according to a known deviation pattern relative to the sensor data that would be received by the wearable device if the wearable device were in a neutral state; and
based on the determined deviated state, determining the second transformation to the first set of sensor data, the determined second transformation to the first set of sensor data being configured produce the mapped sensor data, which is more similar to the sensor data that would be received by the wearable device if the wearable device were in the neutral state than is the first set of sensor data.

### Claim 18 (depends on 13)

The computer-implemented method of claim 13, wherein the second transformation comprises a rotation, a translation, a projection, and/or a scaling, so that the first set of sensor data, as transformed to the mapped sensor data, is more similar to sensor data that would be received by the wearable device if the wearable device were in a neutral state.

### Claim 19 (depends on 13)

The computer-implemented method of claim 13, wherein the ML model includes a first ML model to infer an IMU gesture inference based on the motion data, and a second ML model to infer a biopotential gesture inference based on the biopotential data and the motion data, and the ML model determines the gesture inference based on the IMU gesture inference and the biopotential gesture inference.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, wherein the computer-implemented method further includes:
storing successive biopotential gesture inferences;
determining a threshold number of the successive biopotential gesture inferences have been stored; and
determining the gesture inference based on the threshold number of the successive biopotential gesture inferences and probability information of a confusion matrix, the confusion matrix having been generated during training of the second ML model.