# US 11,998,281 B1 · Extended-reality operating room for robotic surgery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,998,281 B1
- Document ID: US11998281B1
- Status: Granted
- Kind: B1
- Application number: 18/182,130
- Filed: 2023-03-10
- Granted: 2024-06-04
- Earliest claimed: 2023-03-10
- Assignees: IX Innovation LLC
- Applicants: IX Innovation LLC
- Inventors: Jeffrey Roh; Justin Esterberg; John Cronin; Seth Cronin; Michael John Baker
- Web page: https://patentgrep.com/patent/US11998281B1

## CPC

- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/25: {User interfaces for surgical systems}
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/67: for remote operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- G16H 70/20: relating to practices or guidelines
- G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
- A61B 2034/104: {Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2090/306: {using optical fibres}
- A61B 2090/309: {using white LEDs}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Methods, apparatuses, and systems for performing robotic surgery in an extended reality (XR) collaborative customizable virtual operating room are disclosed. The disclosed systems provide a virtual environment in which a surgical robot network receives medical images of a patient and creates a digital twin from a patient's medical images. The surgical robot network allows a first user to create a virtual environment to perform a surgical procedure, select workflow objects, and perform actions on the digital twin. The data of the workflow objects and actions in relation to the digital twin is stored. The first user invites a second user to join the virtual environment who may collaborate with the first user on the workflow objects and actions performed to adjust the workflow, workflow objects, and actions performed. The workflow, workflow objects, actions in relation to the digital twin are sent to a surgical robot.

## Claims

### Claim 1

A computer-implemented method for design of an immersive experiential extended-reality (XR) environment for a robotic surgical procedure, the method comprising:
generating the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,
wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;
generating, using a machine learning model, a robotic process for performing the surgical procedure,
wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;
displaying the immersive experiential XR environment to the first user on a first VR headset and the second user on a second VR headset to allow viewing of,
the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and
one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;
receiving a selection of at least two objects,
wherein the at least two objects are displayed in the immersive experiential XR environment;
determining that the virtual surgical steps have been performed on the 3D digital twin by the one or more actions;
adjusting the robotic process based on the virtual surgical steps; and
transmitting the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,
wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps,
wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the robotic process comprises:
determining a surgery type for the surgical procedure;
determining a correlation coefficient for a data entry corresponding to the surgery type;
based on determining that the correlation coefficient is greater than a threshold correlation coefficient, determining a highest correlated data entry; and
generating the robotic process based on the highest correlated data entry.

### Claim 3 (depends on 1)

The method of claim 1, comprising displaying the adjusted robotic process on a heads-up display.

### Claim 4 (depends on 1)

The method of claim 1, comprising:
detecting an abnormality in the surgical procedure; and
generating a warning based on the abnormality for display in the immersive experiential XR environment.

### Claim 5 (depends on 1)

The method of claim 1, wherein transmitting the adjusted robotic process comprises:
identifying a sensitive body part; and
transmitting an indication of the sensitive body part to the at least one surgical robot.

### Claim 6 (depends on 1)

The method of claim 1, wherein displaying the 3D digital twin comprises:
extracting imaging data corresponding to the patient from a patient database, wherein the imaging data comprises a cross-sectional magnetic resonance imaging image; and
generating the 3D digital twin based on the imaging data.

### Claim 7 (depends on 1)

The method of claim 1, comprising:
retrieving historical data associated with a historical surgical procedure;
generating a color code to distinguish between the historical surgical procedure and the surgical procedure; and
based on the historical data and the color code, displaying the historical surgical procedure and the surgical procedure simultaneously.

### Claim 8 (depends on 1)

The method of claim 1, wherein adjusting the robotic process based on the virtual surgical steps comprises:
determining that at least one of the first user or the second user made an adjustment to the virtual surgical steps, wherein the adjustment comprises a surgical step that differs from a step in the surgical plan;
determining an identifier of the at least one of the first user or the second user; and
storing the adjustment and the identifier in a surgery database.

### Claim 9

A robotic surgical system for design of an immersive experiential XR environment for a robotic surgical procedure, comprising:
a non-transitory, computer-readable storage medium storing computer instructions, which when executed by one or more computer processors cause the robotic surgical system to:
generate the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,
wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;
generate, using a machine learning model, a robotic process for performing the surgical procedure,
wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;
display the immersive experiential XR environment to the first user on a first VR headset and a second user on the second VR headset to allow viewing of,
the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and
one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;
receive a selection of at least two objects,
wherein the at least two objects are displayed in the immersive experiential XR environment for viewing by the first user and the second user;
determine that the virtual surgical steps have been performed on the 3D digital twin by the one or more actions;
adjust the robotic process based on the virtual surgical steps; and
transmit the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,
wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps, and
wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

### Claim 10 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions for generating the robotic process cause the robotic surgical system to:
determine a surgery type for the surgical procedure;
determine a correlation coefficient for a data entry corresponding to the surgery type;
based on determining that the correlation coefficient is greater than a threshold correlation coefficient, determine a highest correlated data entry; and
generate the robotic process based on the highest correlated data entry.

### Claim 11 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions for generating the robotic process cause the robotic surgical system to display the adjusted robotic process on a heads-up display.

### Claim 12 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions cause the robotic surgical system to:
detect an abnormality in the surgical procedure; and
generate a warning based on the abnormality for display in the immersive experiential XR environment.

### Claim 13 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions for transmitting the adjusted robotic process cause the robotic surgical system to:
identify a sensitive body part; and
transmit an indication of the sensitive body part to the at least one surgical robot.

### Claim 14 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions for displaying the 3D digital twin cause the robotic surgical system to:
extract imaging data corresponding to the patient from a patient database, wherein the imaging data comprises a cross-sectional magnetic resonance imaging image; and
generate the 3D digital twin based on the imaging data.

### Claim 15 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions cause the robotic surgical system to:
retrieve historical data associated with a historical surgical procedure;
generate a color code to distinguish between the historical surgical procedure and the surgical procedure; and
based on the historical data and the color code, display the historical surgical procedure and the surgical procedure simultaneously.

### Claim 16 (depends on 9)

The robotic surgical system of claim 9, wherein the computer instructions for adjusting the robotic process based on the virtual surgical steps cause the robotic surgical system to:
determine that at least one of the first user or the second user made an adjustment to the virtual surgical steps, wherein the adjustment comprises a surgical step that differs from a step in the surgical plan;
determine an identifier of the at least one of the first user or the second user; and
store the adjustment and the identifier in a surgery database.

### Claim 17

A surgical robot for design of an immersive experiential XR environment for a robotic surgical procedure, comprising:
one or more computer processors; and
a non-transitory, computer-readable medium storing computer instructions, which when executed by the one or more computer processors cause the surgical robot to:
generate the immersive experiential XR environment by associating at least one virtual model of at least one or more surgical tools and at least one surgical robot with at least one image of a patient,
wherein the immersive experiential XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient and a simulation of a physical presence of a first user immersively collaborating with a second user for simulating the surgical procedure;
generate, using a machine learning model, a robotic process for performing the surgical procedure,
wherein the robotic process is generated based on features extracted from the at least one image of the patient and a surgical plan for the surgical procedure;
display, via an electronic display, the 3D digital twin within the immersive experiential XR environment to the first user on a first VR headset and the second user on a second VR headset to allow viewing of,
the simulation of the physical presence of the first user immersively collaborating with the second user and one or more actions performed by the first user or the second user in the immersive experiential XR environment, and
one or more virtual surgical steps performed on the 3D digital twin according to the surgical plan;
receive a selection of at least two objects,
wherein the at least two objects are displayed in the immersive experiential XR environment for viewing by the first user the second user;
determine that virtual surgical steps have been performed on the 3D digital twin by the one or more actions;
adjust the robotic process based on the virtual surgical steps; and
transmit the adjusted robotic process to the at least one surgical robot to configure the at least one surgical robot with the adjusted robotic process,
wherein the adjusted robotic process comprises the at least two objects and information describing the virtual surgical steps, and
wherein the at least one surgical robot is configured to perform the robotic surgical procedure on the patient according to the adjusted robotic process and monitor the adjusted robotic process being performed by the at least one surgical robot.

### Claim 18 (depends on 17)

The surgical robot of claim 17, wherein the computer instructions for generating the robotic process cause the surgical robot to:
determine a surgery type for the surgical procedure;
determine a correlation coefficient for a data entry corresponding to the surgery type;
based on determining that the correlation coefficient is greater than a threshold correlation coefficient, determine a highest correlated data entry; and
generate the robotic process based on the highest correlated data entry.

### Claim 19 (depends on 17)

The surgical robot of claim 17, wherein the computer instructions for generating the robotic process cause the surgical robot to display the adjusted robotic process on a heads-up display.

### Claim 20 (depends on 17)

The surgical robot of claim 17, wherein the computer instructions cause the surgical robot to:
detect an abnormality in the surgical procedure; and
generate a warning based on the abnormality for display in the immersive experiential XR environment.