Extended-reality operating room for robotic surgery

A robotic surgical system for design of an immersive experiential XR environment for a robotic surgical procedure, comprising:

• a non-transitory, computer-readable storage medium storing computer instructions, which when executed by one or more computer processors cause the robotic surgical system to:
• generate the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,
• wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;
• generate, using a machine learning model, a robotic process for performing the surgical procedure,
• wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;
• display the immersive experiential XR environment to the first user on a first VR headset and a second user on the second VR headset to allow viewing of,
• the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and
• one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;
• receive a selection of at least two objects,
• wherein the at least two objects are displayed in the immersive experiential XR environment for viewing by the first user and the second user;
• determine that the virtual surgical steps have been performed on the 3D digital twin by the one or more actions;
• adjust the robotic process based on the virtual surgical steps;
• and
• transmit the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,
• wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps, and
• wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

A surgical robot for design of an immersive experiential XR environment for a robotic surgical procedure, comprising:

• one or more computer processors;
• and
• a non-transitory, computer-readable medium storing computer instructions, which when executed by the one or more computer processors cause the surgical robot to:
• generate the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,
• wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;
• generate, using a machine learning model, a robotic process for performing the surgical procedure,
• wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;
• display, via an electronic display, the 3D digital twin within the immersive experiential XR environment to the first user on a first VR headset and the second user on a second VR headset to allow viewing of,
• the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and
• one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;
• receive a selection of at least two objects,
• wherein the at least two objects are displayed in the immersive experiential XR environment for viewing by the first user the second user;
• determine that virtual surgical steps have been performed on the 3D digital twin by the one or more actions;
• adjust the robotic process based on the virtual surgical steps;
• and
• transmit the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,
• wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps, and
• wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.