Method and apparatus for virtual training based on tangible interaction

An apparatus for virtual training, comprising:

• an interface device configured to receive data for virtual training from a multi-sensor;
• and
• a processor configured to provide virtual training content based on the data input through the interface device,
• wherein the processor is configured to perform operations by:
• acquiring, through the interface device, data for virtual training, and acquiring a three-dimensional position of a real object based on a depth image and color image of the real object and infrared (IR) data included in the obtained data;
• matching the three-dimensional position of the real object with a position of a virtual object to be applied to a virtual environment;
• performing virtualization of an overall appearance of a user by extracting a depth from depth information on a user image included in the obtained data and matching the extracted depth with the color information;
• visualizing depth data and color data for the user obtained according to virtualization of the user in virtual training content;
• performing correction on joint information using the joint information and the depth information included in the obtained data, and estimating a posture of the user using the corrected joint information;
• and
• estimating a posture of a training tool using the depth information and IR data included in the obtained data;
• wherein the performing of correction on joint information comprises obtaining final joint positions by applying sample points selected from the depth data for the user obtained according to the virtualization of the user and joint positions obtained from the joint information to a distance probability model, and estimating the posture of the user using the obtained final joint positions, and
• wherein the applying the sample points to the distance probability model includes calculating a probability value based at least in part on a distance between a previous joint position and a sample point.