# US 2024/0265915 A1 · System and method for artificial intelligence (ai) assisted activity training

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0265915 A1
- Document ID: US20240265915A1
- Status: Published application
- Kind: A1
- Application number: 18/639,988
- Filed: 2024-04-19
- Published: 2024-08-08
- Earliest claimed: 2021-09-06
- Assignees: Rajiv Trehan
- Inventors: RAJIV TREHAN
- Web page: https://patentgrep.com/patent/US20240265915A1

## CPC

- G10L 15/18: using natural language modelling
- A63B 24/0006: {Computerised comparison for qualitative assessment of motion sequences or the course of a movement}
- A63B 24/0062: {Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance}
- A63B 24/0075: {Means for generating exercise programmes or schemes, e.g. computerized virtual trainer, e.g. using expert databases}
- A63B 24/0087: {Electric or electronic controls for exercising apparatus of groups A63B21/00 - A63B23/00, e.g. controlling load}
- A63B 71/0622: {Visual, audio or audio-visual systems for entertaining, instructing or motivating the user}
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06N 20/00: Machine learning
- G06V 10/82: using neural networks
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G16H 20/30: relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A63B 2024/0009: {Computerised real time comparison with previous movements or motion sequences of the user}
- A63B 2024/0015: {Comparing movements or motion sequences with computerised simulations of movements or motion sequences, e.g. for generating an ideal template as reference to be achieved by the user}
- A63B 2024/0071: {Distinction between different activities, movements, or kind of sports performed}
- A63B 2024/0096: {using performance related parameters for controlling electronic or video games or avatars}
- A63B 2071/0627: {when used improperly, e.g. by giving a warning}
- A63B 2220/806: Video cameras
- A63B 2225/12: with mirrors
- G10L 15/005: {Language recognition}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The disclosure relates to system and method for AI assisted activity training. The method includes presenting a plurality of activity categories to a user and receiving a voice-based input from the user. The method uses an Natural Language Processing model to process received voice-based input to extract selection activity and activity attribute. Contemporaneous to receiving voice-based input, method presents multimedia content in conformance with activity and activity attribute. In response to initiation of the multimedia content, method detects initiation of user activity performance. The method captures video of user activity, overlays, by a smart mirror a pose skeletal model corresponding to user activity performance over reflection of user on smart mirror and process video using AI model to extract user performance parameters. Feedback may be generated based on overlaid pose skeletal model and differential between user performance parameters and target set of performance parameters.

## Claims

### Claim 1

A method for Artificial Intelligence (AI) assisted activity training, the method comprising:
presenting, by a rendering device, a plurality of activity categories to a user, wherein each of the plurality of activity categories comprises a plurality of activities, and wherein the plurality of activity categories are presented as multimedia content;
receiving a voice-based input and a secondary input from the user, wherein the voice-based input comprises an activity training plan comprising a selection of at least one activity from at least one of the plurality of activity categories and at least one activity attribute associated with each of the at least one activity, and wherein the voice-based input is in a source language and the secondary input comprises at least one of the air gestures, biometric inputs, game controllers, input via a keyboard, mouse or any other input;
processing, by a Natural Language Processing (NLP) model, the received voice-based input to extract the selection of at least one activity and the at least one activity attribute, wherein the NLP model is configured using a single language, and wherein the single language is an intermediate language;
initiating, contemporaneous to receiving the voice-based input and the secondary input, presentation of a multimedia content in conformance with the at least one activity and the at least one activity attribute, wherein the multimedia content comprises a plurality of guidance steps performed by a virtual assistant corresponding to the at least one activity;
detecting, via at least one camera, initiation of a user activity performance of the user in response to initiation of the multimedia content, wherein the user activity performance of the user at a given time comprises imitation of one of the at least one activity;
capturing, via the at least one camera, a video of the user activity performance of the user, wherein the at least one camera is placed at distributed locations;
overlaying, by a smart mirror configured with the at least one camera, a pose skeletal model corresponding to the user activity performance over a reflection of the user on the smart mirror, based on the user activity performance of the user, wherein the pose skeletal model comprises skeletal points overlaid on corresponding joints of the user;
processing, in near real time, by an Artificial Intelligence (AI) model, the video to extract a set of user performance parameters of the user based on the user activity performance,
wherein the AI model further comprises a set of portions of AI models configured to perform distinct functionalities for enhanced data security; and
wherein the set of user performance parameters comprises speed of a current activity performance, number of repetitions completed, overall completion of an activity circuit, third-party smart device information, pulse rate of the user, blood pressure of the user and motion of the user;
generating, by the AI model, a near real time feedback during the user activity performance, based on the overlaid pose skeletal model and the differential between the set of user performance parameters and a target set of performance parameters, wherein the target set of performance parameters comprises speed of the target activity performance, blood pressure, target number of repetitions, target pulse rate of the user and target motion of the user; and
rendering, contemporaneous to the user activity performance, the near real time feedback to the user in at least one of an aural form, a visual form, or as haptic feedback, wherein the near real time feedback in the visual form is provided via the post skeletal mode.

### Claim 2 (depends on 1)

The method of claim 1, wherein a first portion of the set of portions of the AI model is configured to perform a pose matching.

### Claim 3 (depends on 2)

The method of claim 2, wherein a second portion of the set of portions of the AI model is configured to perform a skeletal point recognition.

### Claim 4 (depends on 1)

The method of claim 1, further comprising displaying the virtual assistant via a Graphical User Interface (GUI) to provide the near real time feedback to the user.

### Claim 5 (depends on 4)

The method of claim 4, wherein the virtual assistant corresponds to a three-dimensional avatar of a trainer or an expert which performs the plurality of guidance steps corresponding to the at least one activity.

### Claim 6 (depends on 1)

The method of claim 1, wherein a display of the plurality of guidance steps and related information is adjusted, based on at least one of eye motion, a voice-based input, a hand gesture, and a position of the user to place the plurality of guidance steps and the related information appropriately relative to a viewing angle and a current position of the user.

### Claim 7 (depends on 1)

The method of claim 1, wherein the near real time feedback in the visual form further comprises instruction in a textual form and graphic elements, wherein the graphic elements further comprise dynamic elements for motion based graphics and static elements.

### Claim 8 (depends on 1)

The method of claim 1, wherein the near real time feedback further comprises a first type of vibration indicative of incorrect performance, and a second type of vibration indicative of a correct performance, wherein the first type of vibration corresponds to a longer duration as compared to the second type of vibration having a shorter duration in bursts.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
rendering, on the smart mirror, a live training session conducted by a human trainer.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
integrating the smart mirror with a plurality of social media applications to connect users of other smart mirrors with the user.

### Claim 11 (depends on 1)

The method of claim 1, wherein a priority of messages associated with the near real time feedback is handled based on priority queues.
