Method and system for generating a training dataset for keypoint detection, and method and system for predicting 3D locations of virtual markers on a marker-less subject

A system for generating a training dataset for keypoint detection, the system comprising:

• an optical marker-based motion capture system configured to capture a plurality of markers over a period of time, wherein each marker is placed on a bone landmark of a human or animal subject or a keypoint of an object, and is captured as a 3D trajectory;
• a plurality of colour video cameras configured to capture the human or animal subject or the object over the period of time as sequences of 2D images, wherein the optical marker-based motion capture system and the plurality of colour video cameras are calibrated;
• a computer configured to:
• receive the sequences of 2D images captured by the plurality of colour video cameras and the 3D trajectories captured by the optical marker-based motion capture system;
• and
• a synchronization pulse generator in communication with the optical marker-based motion capture system and the plurality of colour video cameras,
• wherein the computer is further configured to:
• for each marker, project the 3D trajectory to each of the 2D images to determine a 2D location in each 2D image;
• based on the determined 2D locations of each marker in the sequences of 2D images and an exposure-related time of the plurality of colour video cameras, interpolate a 3D position of each marker projected to each of the 2D images;
• based on the interpolated 3D positions of the plurality of markers and an extended volume derived from two or more of the markers having an anatomical relationship or functional relationship with one another, generate a 2D bounding box around the human or animal subject or the object in each 2D image, wherein the extended volume is obtainable by extending coverage of each of the two or more of the markers by different amounts to a point covering a part of the human or animal subject or the object that is adjacent to the bone landmark or keypoint on which each of the two or more of the markers is placed;
• and
• generate the training dataset comprising at least one 2D image selected from the sequences of 2D images, the determined 2D location of each marker in the selected at least one 2D image, and the generated 2D bounding box for the selected at least one 2D image, and
• the synchronization pulse generator is configured to receive a synchronization signal from the optical marker-based motion capture system for coordinating the human or animal subject or the object to be simultaneously captured by the plurality of colour video cameras.