Systems and methods for computer vision and machine-learning based form feedback

A method implemented by a computing system including a plurality of three-dimensional (3D) motion sensors, the plurality of 3D motion sensors including one or more infrared (IR) sensors and one or more red, green, and blue (RGB) sensors, the plurality of 3D motion sensors being capable of capturing IR video, depth video, and RGB images, and one or more processors and storage media storing machine-readable instructions, wherein the method is performed using the one or more processors, the method comprising:

• periodically emitting, by at least a portion of the plurality of 3D motion sensors, IR pulses;
• capturing, using the plurality of 3D motion sensors and at least partially based on the periodically emitted IR pulses, IR video of a user and exercise equipment being used by the user through a plurality of movements of an exercise, depth video of the user and the exercise equipment being used by the user through the plurality of movements of the exercise, and RGB images of the user and the exercise equipment being used by the user through the plurality of movements of the exercise;
• generating, at least partially based on the captured IR pulses, a point cloud of the user and the exercise equipment being used by the user through the plurality of movements of the exercise;
• generating, in response to generating the point cloud of the user and the exercise equipment, a 3D model of the user, the 3D model including the point cloud of the user and the exercise equipment;
• inputting the IR video of the user, the depth video of the user, and the RGB images of the user into a machine learning model;
• estimating a set of joints of the user in the 3D model, the estimating using the point cloud and the machine learning model with the inputted IR video, the inputted depth video, and the inputted RGB images;
• tracking, based on the estimated set of joints of the user in the 3D model, the user's motion over a period of time;
• determining, based on the tracked user's motion over the period of time and one or more exercise models, a number of repetitions of the exercise performed by the user over the period of time;
• determining, based on the tracked user's motion over the period of time and one or more form feedback models, a form feedback value from a set of form feedback values, the set of form feedback values including a satisfactory value and an unsatisfactory value;
• calculating, based on the number of repetitions and the form feedback value, a user exercise score;
• providing, via a graphical user interface, the user exercise score and the form feedback value to the user, thereby instructing the user to adjust their form during subsequent repetitions of the exercise.