System and method for virtual reality training, simulation of a virtual robotic surgery environment

A method for simulating a virtual robotic surgery environment comprising one or more virtual robotic arms each coupled to a virtual surgical instrument at its distal end, a virtual operating table, and a virtual patient model representing a virtual patient lying on top of the virtual operating table, whereby the one or more virtual robotic arms are arranged along the virtual operating table, the method comprising:

• receiving, using an input device (202), an input from an operator (204), wherein the input device (202) comprises at least one hand controller for each hand or is configured to receive hand gestures of the operator (204), and wherein the input device (202) is tracked using at least one of infra-red tracking, optical tracking using image processing, radio frequency tracking, or inertial measurement unit (IMU) sensor tracking;
• storing, using a server (208), in a database (210) at least one of a diagnostic scan and patient details for one or more patients or a virtual tutorial for one or more robotic surgical procedures;
• extracting, using a processor (206), a relevant data (212) based on the received input, from the database (210) stored on the server (208), wherein extracting the relevant data (212) comprises fetching, based on the received input, at least one of a three-dimensional (3D) model generated from diagnostic imaging data, patient details of one or more patients, or the virtual tutorial for one or more robotic surgical procedures;
• retrieving, from the database (210) stored on the server (208), patient-specific diagnostic imaging data associated with the virtual patient model, wherein the patient-specific diagnostic imaging data comprises a plurality of Digital Imaging and Communications in Medicine (DICOM) files generated from layer-by-layer computed tomography (CT), magnetic resonance imaging (MRI), or ultrasound scanning of the patient, each DICOM file corresponding to a cross-sectional layer of patient anatomy;
• converting, using the server (208), the retrieved patient-specific diagnostic imaging data comprising one or more two-dimensional (2D) diagnostic scans into a three-dimensional (3D) anatomical model by applying segmentation logic, wherein the 3D anatomical model includes an anatomical organ corresponding to the virtual patient model, and wherein the segmentation logic comprises:
• mapping the plurality of DICOM files into a correct layer sequence;
• thresholding pixel values of each layer to identify and contour anatomical structures of interest;
• stacking the contoured layers according to the layer mapping sequence to construct volumetric data;
• and
• generating the 3D anatomical model based on a mean volumetric value derived from the stacked pixel data;
• storing, using the server (208), the 3D anatomical model as virtual patient anatomy data;
• generating, using the processor (206), an augmented three-dimensional (3D) model or a three-dimensional (3D) holographic projection of the 3D anatomical model for rendering on a stereoscopic display (214);
• rendering, using the processor (206), the virtual robotic arms, the virtual surgical instruments, the virtual operating table, and the virtual patient model on the stereoscopic display (214), wherein the stereoscopic display (214) is coupled to a virtual reality headset or a mixed reality headset, and wherein the virtual robotic arms, the virtual surgical instruments, the virtual operating table, and the virtual patient model are rendered based on predetermined models thereof;
• manipulating, using the processor (206), based on another input received from the operator (204), the augmented 3D model or the 3D holographic projection within the virtual robotic surgery environment, wherein the manipulation comprises at least one of enlarging, rotating, filtering, slicing, or adjusting the scale of the augmented 3D model or the 3D holographic projection;
• superimposing, using the processor (206), the 3D anatomical model corresponding to the anatomical organ, or the augmented 3D model or the 3D holographic projection thereof, onto the virtual patient model within the virtual robotic surgery environment at a one-to-one (1:1) scale relative to the virtual patient model;
• and
• enabling, during a simulated surgical procedure, identification of a position and orientation of the anatomical organ relative to the virtual patient model based on the superimposed 3D anatomical model, the augmented 3D model, or the 3D holographic projection to enable pre-operative planning of the surgical procedure within the virtual robotic surgery environment.