# US 2022/0036052 A1 · Detection of kinetic events and mechanical variables from uncalibrated video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036052 A1
- Document ID: US20220036052A1
- Status: Published application
- Kind: A1
- Application number: 17/389,848
- Filed: 2021-07-30
- Published: 2022-02-03
- Earliest claimed: 2020-07-31
- Assignees: QualiaOS, Inc.
- Applicants: QualiaOS, Inc.
- Inventors: Kevin John Prince; Carlos Dietrich; Justin Ali Kennedy
- Web page: https://patentgrep.com/patent/US20220036052A1

## CPC

- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- A63B 24/0006: {Computerised comparison for qualitative assessment of motion sequences or the course of a movement}
- G06T 7/215: Motion-based segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- G06V 20/42: {of sport video content}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and techniques are provided to identify, analyze, and evaluate key events and mechanical variables in videos of human motion related to an action, such as may be used in training for various sports and other activities. Information about the action is calculated based on analysis of the video such as via keypoint identification, pose identification and/or estimation, and related calculations, and provided automatically to the user to allow for improvement of the action.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving an uncalibrated video showing performance of an action by a human subject;
extracting a plurality of two-dimensional (2D) poses of the human subject while performing the action;
based upon the 2D poses, detecting one or more key events in the uncalibrated video, each of the key events corresponding to a predefined portion of the action;
based upon the key events, computing one or more mechanical variables of the human subject performing the action; and
based upon the one or more mechanical variables, automatically providing information about performance of the action by the human subject, the information indicating at least one portion of the action that was performed sub-optimally by the human subject.

### Claim 2 (depends on 1)

The method of claim 1, wherein the extracting the plurality of 2D poses comprises, for each 2D pose of the plurality of 2D poses:
identifying an arrangement of keypoints on the human subject in one or more frames of the uncalibrated video;
comparing the arrangement of keypoints to a library of predefined two-dimensional (2D) signatures for the action; and
based on the comparing, selecting the 2D pose from the library.

### Claim 3 (depends on 2)

the method of claim 2, wherein each 2D signature in the library corresponds to a 2D projection of the action as seen from a corresponding point of view.

### Claim 4 (depends on 2)

The method of claim 2, wherein the 2D signatures are based upon a plurality of three-dimensional (3D) motion patterns previously obtained from 3D seed data.

### Claim 5 (depends on 4)

The method of claim 4, wherein the 3D seed data comprises motion capture data.

### Claim 6 (depends on 4)

The method of claim 4, wherein the 3D seed data comprises simulated motion data.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
prior to receiving the uncalibrated video, generating a library of predefined 2D signatures for the action based upon a plurality of 3D motion patterns obtained from 3D seed data.

### Claim 8 (depends on 1)

The method of claim 1, wherein the information further comprises an identification of one or more exercises, drills, or activities to perform to cause an improvement in the portion of the action that was performed sub-optimally by the human subject

### Claim 9 (depends on 1)

The method of claim 1, wherein the action is a pitch by a baseball pitcher.

### Claim 10 (depends on 9)

The method of claim 9, wherein the key events comprise one or more selected from the group consisting of: a front foot lift, a max leg lift, a foot strike, a max hip and shoulder separation, shoulders squared up to target, and ball release.

### Claim 11

A system comprising:
a processor configured to:
receive an uncalibrated video showing performance of an action by a human subject;
extract a plurality of two-dimensional (2D) poses of the human subject while performing the action;
based upon the 2D poses, detect one or more key events in the uncalibrated video, each of the key events corresponding to a predefined portion of the action;
based upon the key events, compute one or more mechanical variables of the human subject performing the action; and
based upon the one or more mechanical variables, automatically determine information about performance of the action by the human subject, the information indicating at least one portion of the action that was performed sub-optimally by the human subject; and
a user interface comprising a display, the interface capable of providing the information about the performance of the action by the human subject.

### Claim 12 (depends on 11)

The system of claim 11, wherein extracting the plurality of 2D poses comprises, for each 2D pose of the plurality of 2D poses:
identifying an arrangement of keypoints on the human subject in one or more frames of the uncalibrated video;
comparing the arrangement of keypoints to a library of predefined two-dimensional (2D) signatures for the action; and
based on the comparing, selecting the 2D pose from the library.

### Claim 13 (depends on 11)

The system of claim 11, further comprising a computerized video capture device configured to capture the uncalibrated video.

### Claim 14 (depends on 12)

The system of claim 12, wherein the video capture device comprises a phone or tablet with an integrated camera.

### Claim 15 (depends on 11)

The system of claim 11, wherein the information further comprises an identification of one or more exercises, drills, or activities to perform to cause an improvement in the portion of the action that was performed sub-optimally by the human subject

### Claim 16 (depends on 11)

The system of claim 11, wherein the action is a pitch by a baseball pitcher.

### Claim 17

A non-transitory computer-readable medium storing a plurality of instructions which, when executed by a processor, cause the processor to:
receive an uncalibrated video showing performance of an action by a human subject;
extract a plurality of two-dimensional (2D) poses of the human subject while performing the action;
based upon the 2D poses, detect one or more key events in the uncalibrated video, each of the key events corresponding to a predefined portion of the action;
based upon the key events, compute one or more mechanical variables of the human subject performing the action; and
based upon the one or more mechanical variables, automatically determine information about performance of the action by the human subject, the information indicating at least one portion of the action that was performed sub-optimally by the human subject; and
a user interface comprising a display, the interface capable of providing the information about the performance of the action by the human subject.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein extracting the plurality of 2D poses comprises, for each 2D pose of the plurality of 2D poses:
identifying an arrangement of keypoints on the human subject in one or more frames of the uncalibrated video;
comparing the arrangement of keypoints to a library of predefined two-dimensional (2D) signatures for the action; and
based on the comparing, selecting the 2D pose from the library.
