# US 2021/0174700 A1 · Interactive training of body-eye coordination and reaction times using multiple mobile device cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174700 A1
- Document ID: US20210174700A1
- Status: Published application
- Kind: A1
- Application number: 17/179,435
- Filed: 2021-02-19
- Published: 2021-06-10
- Earliest claimed: 2018-12-11
- Assignees: NEX Team Inc.
- Applicants: NEX Team Inc.
- Inventors: Qi Zhang; Arron Mollet; Wing Hung Chan; Keng Fai Lee
- Web page: https://patentgrep.com/patent/US20210174700A1

## CPC

- A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)}
- A61B 5/1116: {Determining posture transitions}
- A61B 5/486: {Biofeedback (using electroencephalography [EEG] A61B5/375)}
- A63B 24/0062: {Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance}
- A63B 24/0075: {Means for generating exercise programmes or schemes, e.g. computerized virtual trainer, e.g. using expert databases}
- A63B 71/0622: {Visual, audio or audio-visual systems for entertaining, instructing or motivating the user}
- A63B 71/0669: {Score-keepers or score display devices}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G09B 19/0038: {Sports}
- A61B 2503/10: Athletes
- A61B 2505/09: Rehabilitation or training
- A63B 69/0071: {for basketball}
- A63B 2071/063: {Spoken or verbal instructions}
- A63B 2071/0694: {Visual indication, e.g. Indicia}
- A63B 2220/806: Video cameras
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Methods and systems for multi-player training of body-eye coordination using mobile computing devices each having a camera are provided. The methods and systems execute instructions to capture a first training video of a first of player, using a first camera on a first mobile computing device; superimpose a visual cue onto the first training video at a visual cue location and for a cue period starting from a first time instant; determine whether the first player has responded to the visual cue at a second time instant within the cue period; receive a notification from a second mobile computing device, where the notification was generated in response to determining that a second player has responded to the visual cue at a third time instant within the cue period; and in response to a determination that the first player has responded to the visual cue and to the notification, generate a feedback to the first player.

## Claims

### Claim 1

A non-transitory storage medium, the non-transitory storage medium storing executable instructions, the executable instructions when executed by a hardware processor causes the hardware processor to execute a process for facilitating multi-player training of body-eye coordination using mobile computing devices each having a camera, the non-transitory storage medium comprising executable instructions to:
capture a first training video of a first of player, using a first camera on a first mobile computing device;
superimpose a visual cue onto the first training video at a visual cue location and for a cue period starting from a first time instant;
determine whether the first player has responded to the visual cue at a second time instant within the cue period, by analyzing a body posture flow of the first player between the first time instant and the second time instant, wherein the body posture flow of the first player is extracted from the first training video by performing a computer vision algorithm on one or more frames of the first training video;
receive a notification from a second mobile computing device, wherein the notification was generated in response to determining that a second player has responded to the visual cue at a third time instant within the cue period; and
in response to a determination that the first player has responded to the visual cue and to the notification, generate a feedback to the first player.

### Claim 2 (depends on 1)

The non-transitory storage medium of claim 1, wherein the determination that the second player has responded to the visual cue at the third time instant within the cue period comprises executable instructions to analyze a body posture flow of the second player between the first time instant and the third time instant, wherein the body posture flow of the second player was extracted from a second training video of the second player by performing the computer vision algorithm on one or more frames of the second training video, wherein the second training video was captured using a second camera on the second mobile computing device, and wherein the visual cue was superimposed onto the second training video at the visual cue location starting from the first time instant.

### Claim 3 (depends on 1)

The non-transitory storage medium of claim 1, further comprising executable instructions to:
generate the feedback based on an identity of at least one player that responded.

### Claim 4 (depends on 1)

The non-transitory storage medium of claim 1, wherein the visual cue is a symbol superimposed onto the first training video at the first location of an image plane of the first training video.

### Claim 5 (depends on 1)

The non-transitory storage medium of claim 1, wherein the determination whether one of the players has responded to the visual cue comprises determining a player movement to virtually touch the symbol in the image plane with a body part.

### Claim 6 (depends on 1)

The non-transitory storage medium of claim 1, wherein the determination whether one of the players has responded to the visual cue comprises determining whether the at least one player has virtually touched the symbol in the image plane with a sports equipment object.

### Claim 7 (depends on 1)

The non-transitory storage medium of claim 1, wherein the determination whether one of the players has responded to the visual cue comprises determining whether at least one player has performed a predetermined sequence of movements.

### Claim 8 (depends on 1)

The non-transitory storage medium of claim 1, further comprising executable instructions to:
determine a reaction time as a duration between the first time instant and the second time instant, wherein the feedback is a quality score generated based on the reaction time.

### Claim 9 (depends on 1)

The non-transitory storage medium of claim 1, wherein the training comprises a plurality of difficulty levels based at least in part on a duration of the cue period.

### Claim 10 (depends on 1)

The non-transitory storage medium of claim 1,
wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) for detecting one or more key points of the first player in an image plane, and
wherein the CNN module has been trained using one or more prior videos.

### Claim 11 (depends on 1)

The non-transitory storage medium of claim 1,
wherein the training video comprises a dribbling activity, and
wherein the superimposing the first training video with the visual cue is in response to determining that at least one player has dribbled for a predetermined number of times before the first time instant.

### Claim 12 (depends on 1)

The non-transitory storage medium of claim 1, further comprising executable instructions to:
wait for a period of wait time before the superimposing the training video with the visual cue, wherein a duration of the wait time is based on a detected player action during the wait time.

### Claim 13 (depends on 1)

The non-transitory storage medium of claim 1,
wherein the first training video comprises a juggling activity, and
wherein the superimposing the training video with the visual cue is in response to determining that at least one player has juggled for a predetermined number of times before the first time instant.

### Claim 14 (depends on 1)

The non-transitory storage medium of claim 1, further comprising executable instructions to:
generate a training statistic for at least one player based on the training video, wherein the training video comprises a dribbling activity performed by the at least one player, and wherein the training statistic includes at least one of a reaction speed and a dribbling speed.

### Claim 15 (depends on 14)

The non-transitory storage medium of claim 14, wherein the training statistic comprises a first current statistic that is associated with the training video and a second historical statistic that is associated with one or more historical training sessions associated with at least one player.

### Claim 16 (depends on 1)

The non-transitory storage medium of claim 1, wherein the feedback is a score.

### Claim 17 (depends on 1)

The non-transitory storage medium of claim 1, wherein the first training video comprise basketball training activities.

### Claim 18 (depends on 1)

The non-transitory storage medium of claim 1, further comprising executable instructions to:
receive a second notification from a third mobile computing device, wherein the second notification was generated based on a third training video of a third player, captured using a third camera on the third mobile computing device.

### Claim 19

A system comprising a hardware processor and a non-transitory storage medium, the non-transitory storage medium storing executable instructions, the executable instructions when executed by the hardware processor causes the hardware processor to execute a process for facilitating multi-player training of body-eye coordination using mobile computing devices each having a camera, the executable instructions comprising instructions to:
capture a first training video of a first of player, using a first camera on a first mobile computing device;
superimpose a visual cue onto the first training video at a visual cue location and for a cue period starting from a first time instant;
determine whether the first player has responded to the visual cue at a second time instant within the cue period, by analyzing a body posture flow of the first player between the first time instant and the second time instant, wherein the body posture flow of the first player is extracted from the first training video by performing a computer vision algorithm on one or more frames of the first training video;
receive a notification from a second mobile computing device, wherein the notification was generated in response to determining that a second player has responded to the visual cue at a third time instant within the cue period; and
in response to a determination that the first player has responded to the visual cue and to the notification, generate a feedback to the first player.

### Claim 20 (depends on 19)

The system of claim 19, wherein the determination that the second player has responded to the visual cue at the third time instant within the cue period comprises executable instructions to analyze a body posture flow of the second player between the first time instant and the third time instant, wherein the body posture flow of the second player was extracted from a second training video of the second player by performing the computer vision algorithm on one or more frames of the second training video, wherein the second training video was captured using a second camera on the second mobile computing device, and wherein the visual cue was superimposed onto the second training video at the visual cue location starting from the first time instant.
