# US 2020/0401793 A1 · Apparatus and methods for determining multi-subject performance metrics in a three-dimensional space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401793 A1
- Document ID: US20200401793A1
- Status: Published application
- Kind: A1
- Application number: 16/914,232
- Filed: 2020-06-26
- Published: 2020-12-24
- Earliest claimed: 2020-06-26
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Nelson Leung; Jonathan K. Lee; Bridget L. Williams; Sameer Sheorey; Amery Cong; Mehrnaz Khodam Hazrati; Mourad S. Souag; Adam Marek; Pawel Pieniazek; Bogna Bylicka; Jakub Powierza; Anna Banaszczyk-fiszer
- Web page: https://patentgrep.com/patent/US20200401793A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/292: Multi-camera tracking
- G06T 7/596: {from three or more stereo images}
- G06T 7/73: using feature-based methods
- G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19)
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/757: {Matching configurations of points or features}
- G06V 20/42: {of sport video content}
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Apparatus and methods for extraction and calculation of multi-person performance metrics in a three-dimensional space. An example apparatus includes a detector to identify a first subject in a first image captured by a first image capture device based on a first set of two-dimensional kinematic keypoints in the first image, the two-dimensional kinematic keypoints corresponding to a joint of the first subject, the first image capture device associated with a first view of the first subject, a multi-view associator to verify the first subject using the first image and a second image captured by a second image capture device, the second image capture device associated with a second view of the first subject, the second view different than the first view, and a keypoint generator to generate three-dimensional keypoints for the first subject using the first set of two-dimensional kinematic keypoints.

## Claims

### Claim 1

An apparatus, comprising:
a detector to identify a first subject in a first image captured by a first image capture device based on a first set of two-dimensional kinematic keypoints in the first image, the first set of two-dimensional kinematic keypoints corresponding to one or more joints of the first subject, the first image capture device associated with a first view of the first subject;
a multi-view associator to verify the first subject using the first image and a second image captured by a second image capture device, the second image capture device associated with a second view of the first subject, the second view different than the first view;
a keypoint generator to generate three-dimensional keypoints for the first subject using the first set of two-dimensional kinematic keypoints and a second set of keypoints in the second image; and
a biomechanics analyzer to determine a performance metric for the first subject using the three-dimensional keypoints.

### Claim 2 (depends on 1)

The apparatus of claim 1, further including a bounding box generator to generate a first bounding box for the first subject in the first image and a second bounding box for a second subject in the second image, the multi-view associator to identify the first subject in the first image using the first bounding box and in the second image using the second bounding box.

### Claim 3 (depends on 2)

The apparatus of claim 2, further including a tracker to assign a first subject identifier to the first bounding box and a second subject identifier to the second bounding box, the multi-view associator to associate the first subject identifier and the second subject identifier with the first subject.

### Claim 4 (depends on 1)

The apparatus of claim 1, further including an image augmenter to increase a resolution of at least one of the first image or the second image.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein the multi-view associator is to execute a neural network model to associate the first subject identifier and the second subject identifier with the first subject.

### Claim 6

(canceled)

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the first image and the second image each include a second subject, the detector to identify the second subject based on a third set of two-dimensional kinematic keypoints in the first image and a fourth set of two-dimensional kinematic keypoints in the second image.

### Claim 8

(canceled)

### Claim 9

(canceled)

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the detector is to execute a two-dimensional pose estimation algorithm to identify the first set of two-dimensional kinematic keypoints.

### Claim 11

(canceled)

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the performance metric includes one or more of velocity, acceleration, shoulder sway, center of mass, or stride frequency of the first subject.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein the biomechanics analyzer is to assign a first weight to one or more of the three-dimensional keypoints to determine a first performance metric and assign a second weight to the one or more of the three-dimensional keypoints to determine a second performance metric, the second performance metric different than the first performance metric.

### Claim 14

A system comprising:
a first image capture device to generate first image data, the first image data including a first view of a subject;
a second image capture device to generate second image data, the second image data including a second view of the subject; and
a processor to:
predict first positions of two-dimensional keypoints of the subject based on the first image data;
assign a first identifier to the subject in the first image data based on the first positions of the two-dimensional keypoints;
predict second positions of two-dimensional keypoints of the subject based on the second image data;
assign a second identifier to the subject in the second image data based on the second positions of two-dimensional keypoints;
identify the subject as a first subject in the first image data and the second image based on the first identifier and the second identifier;
predict three-dimensional keypoints for the first subject based on the first positions of the two-dimensional keypoints and the second positions of the two-dimensional keypoints in the second image; and
determine a performance metric for the subject using the three-dimensional keypoints.

### Claim 15 (depends on 14)

The system of claim 14, wherein the processor is to predict the first positions of the two-dimensional keypoints of the first subject based on the first image data and the second positions of the two-dimensional keypoints of the first subject based on the second image data substantially concurrently.

### Claim 16 (depends on 15)

The system of claim 15, wherein the processor is to generate a first bounding box for the first subject in the first image and a second bounding box for the second subject in the second image.

### Claim 17 (depends on 16)

The system of claim 16, wherein the processor is to identify the first subject in the first image using the first bounding box and in the second image using the second bounding box.

### Claim 18 (depends on 17)

The system of claim 17, wherein the processor is to assign a first subject identifier to the first bounding box and a second subject identifier to the second bounding box, the processor to associate the first subject identifier and the second subject identifier with the first subject.

### Claim 19 (depends on 18)

The system of claim 18, wherein the processor is to execute a neural network model to associate the first subject identifier and the second subject identifier with the first subject.

### Claim 20

(canceled)

### Claim 21

(canceled)

### Claim 22

(canceled)

### Claim 23

(canceled)

### Claim 24

(canceled)

### Claim 25

(canceled)

### Claim 26

At least one non-transitory computer readable medium comprising instructions that, when executed, cause at least one processor to at least:
identify a first subject in a first image captured by a first image capture device based on a first set of two-dimensional kinematic keypoints in the first image, the first set of two-dimensional kinematic keypoints corresponding to one or more joints of the first subject, the first image capture device associated with a first view of the first subject;
verify the first subject using the first image and a second image captured by a second image capture device, the second image capture device associated with a second view of the first subject, the second view different than the first view;
generate three-dimensional keypoints for the first subject using the first set of two-dimensional kinematic keypoints and a second set of keypoints in the second image; and
determine a performance metric for the first subject using the three-dimensional keypoints.

### Claim 27 (depends on 26)

The at least one non-transitory computer readable medium as defined in claim 26, wherein the instructions, when executed, cause the at least one processor to generate a first bounding box for the first subject in the first image and a second bounding box for a second subject in the second image, and identify the first subject in the first image using the first bounding box and in the second image using the second bounding box.

### Claim 28 (depends on 27)

The at least one non-transitory computer readable medium as defined in claim 27, wherein the instructions, when executed, cause the at least one processor to assign a first subject identifier to the first bounding box and a second subject identifier to the second bounding box, and associate the first subject identifier and the second subject identifier with the first subject.

### Claim 29 (depends on 28)

The at least one non-transitory computer readable medium as defined in claim 28, wherein the instructions, when executed, cause the at least one processor to execute a neural network model to associate the first subject identifier and the second subject identifier with the first subject.

### Claim 30 (depends on 26)

The at least one non-transitory computer readable medium as defined in claim 26, wherein the instructions, when executed, cause the at least one processor to increase a resolution of at least one of the first image or the second image.

### Claim 31

(canceled)