Vision-haptics fused augmented reality simulator for dental surgical skill training

A vision-haptics fused augmented reality simulator for dental surgical skill training, comprising a simulation training platform, a dental operation training system based on a haptic feedback device, and an observation system based on an augmented reality head-mounted display;

• the dental training platform has a shape close to an artificial head phantom used in a dental practice training, is convenient for an user to be familiar with an operating environment, is used as a finger support, and achieves a transfer of a training effect;
• the dental operation training system based on the haptic feedback device fixes a bracket of the haptic feedback device onto the dental training platform and replaces a dental surgical instrument with an end handle of the haptic feedback device so as to be held by the user;
• the user operates the end handle of the haptic feedback device to move, with a motion range covering a dental range of the artificial head phantom;
• the haptic feedback device can generate a specific output force to simulate haptics of touching various tissues comprising tooth, gum, tongue and cheek;
• the observation system based on the augmented reality head-mounted display superposes a virtual environment onto a real world to perform display, receives mesh information, and displays a virtual dental environment constructed by a virtual dental model and a virtual tool model, or observes real environments of the dental training platform and the haptic feedback device, and enhances an immersion feeling in a training process;
• an implementation process of the training simulator:
• a. based on CBCT scan and true color scan data of a patient, as well as size and shape measurement data of the dental surgical instrument, the virtual dental model and the virtual tool model are established to construct the virtual dental environment;
• b. based on the virtual dental model and feature points on the artificial head phantom, a model matching is performed, so that the virtual environment is matched with a training model in spatial positions of corresponding points;
• c. after the model matching is completed, the mesh information is transmitted to an virtual dental surgery simulation algorithm for an virtual dental surgery simulation calculation, the user controls a virtual tool to move through the end handle of the haptic feedback device, and observes from different angles through the augmented reality head-mounted display, posture information of the end handle of the haptic feedback device and position information of an operator's head are acquired in real time by the virtual dental surgery simulation algorithm at a frequency of greater than 1000 Hz and a frequency of greater than 60 Hz, respectively, and after calculation, haptics information and the mesh information of the virtual environment are output;
• d. the mesh information needs to be transmitted to the augmented reality head-mounted display in a wireless transmission mode to perform display, a visual information processing is performed before transmission, to simplify the mesh information, reduce the number of mesh vertices on the premise of ensuring a sharp feature of a mesh model, and increase transmission speed, to realize a refresh frequency of greater than 60 Hz;
• e. based on acquired information of the operator's head, a haptics-vision space calibration matrix is obtained, and a haptics-vision space calibration is performed, so that an output force felt by the user is matched with visual information observed by the user, and after visually observing that the virtual tool touches a dental tissue in the virtual environment, a touch force in a correct direction can be felt at a correct position.