System and method for matching a test frame sequence with a reference frame sequence

A computer system for matching a test frame sequence with a reference frame sequence, the reference frame sequence demonstrating a physical exercise performed by a first human, the system comprising:

• an interface configured to receive a pose data file representing for at least a subset of frames of the reference frame sequence, based on a list of three-dimensional joint coordinates representing positions of a plurality of joints of a three-dimensional skeleton of the first human in the respective frame, a plurality of two-dimensional skeleton projections on a virtual spherical surface with a particular joint of the three-dimensional skeleton at the geometric center of the spherical surface, wherein each two-dimensional skeleton projection on the spherical surface for a particular frame of the subset corresponds to a two-dimensional reference pose image of the three-dimensional skeleton of the first human from a different viewing angle, with the two-dimensional reference pose image being a characteristic pose of the physical exercise, and further configured to receive the test frame sequence representing movements of a second human while imitating the physical exercise, the test frame sequence captured at a particular angle by a camera device;
• a real-time two-dimensional skeleton detector module configured to detect a two-dimensional skeleton of the second human in a current test frame of the test frame sequence, wherein the two-dimensional skeleton of the second human is a two-dimensional representation of the pose of the second human in the respective test frame;
• and
• a pose matching module configured to select a particular two-dimensional skeleton projection of the first human with the minimum mathematical distance from the two-dimensional skeleton of the second human in the current test frame to match the current pose of the second human in the current test frame with a corresponding reference pose representation of the pose data file, the particular two-dimensional skeleton projection representing the corresponding reference pose at a viewing angle which corresponds to the particular angle of the camera device.

A computer program product for matching a test frame sequence with a reference frame sequence, the reference frame sequence demonstrating a physical exercise performed by a first human, wherein the computer program product, when loaded into a memory of a computing device and executed by at least one processor of the computing device, causes the at least one processor to:

• receive a pose data file representing for at least a subset of frames of the reference frame sequence, based on a list of three-dimensional joint coordinates representing positions of a plurality of joints of a three-dimensional skeleton of the first human in the respective frame, a plurality of two-dimensional skeleton projections onto a virtual spherical surface with a particular joint of the three-dimensional skeleton at the geometric center of the spherical surface, wherein each two-dimensional skeleton projection for a particular frame of the subset corresponds to a two-dimensional reference pose image of the three-dimensional skeleton of the first human from a different viewing angle, with the two-dimensional reference pose image being a characteristic pose of the physical exercise;
• receive the test frame sequence representing movements of a second human while imitating the physical exercise, the test frame sequence captured at a particular angle by a camera device;
• detect, with a real-time two-dimensional skeleton detector, a two-dimensional skeleton of the second human in a current test frame of the test frame sequence, wherein the two-dimensional skeleton of the second human is a two-dimensional representation of the pose of the second human in the respective test frame;
• and
• select a particular two-dimensional skeleton projection of the first human with the minimum mathematical distance from the two-dimensional skeleton of the second human in the current test frame to match the current pose of the second human in the current test frame with a corresponding reference pose representation in the pose data file, the particular two-dimensional skeleton projection representing the corresponding reference pose at a viewing angle which corresponds to the particular angle of the camera device.