# US 2021/0241656 A1 · Mixed-reality endoscope and surgical tools with haptic feedback for integrated virtual-reality visual and haptic surgical simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241656 A1
- Document ID: US20210241656A1
- Status: Published application
- Kind: A1
- Application number: 17/048,962
- Filed: 2019-04-18
- Published: 2021-08-05
- Earliest claimed: 2018-04-18
- Assignees: Arizona Board of Regents on Behalf of the University of Arizona
- Applicants: Arizona Board of Regents On Behalf of the University of Arizona
- Inventors: Samuel Barber; Saurabh Jain; Young-Jun Son; Eugene Chang
- Web page: https://patentgrep.com/patent/US20210241656A1

## CPC

- G09B 23/285: {for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas}
- A61B 17/24: for use in the oral cavity, larynx, bronchial passages or nose (for medical inspection of cavities or tubes in the body A61B1/00); Tongue scrapers
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- B33Y 80/00: Products made by additive manufacturing
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/11: Region-based segmentation
- G09B 23/30: Anatomical models {(G09B23/281 - G09B23/288 take precedence)}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 40/63: for local operation
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 17/29: Forceps for use in minimally invasive surgery
- A61B 17/320016: {Endoscopic cutting instruments, e.g. arthroscopes, resectoscopes (A61B10/0233, A61B17/29, A61B17/3207 take precedence)}
- A61B 17/3201: Scissors (for biopsy A61B10/02)
- A61B 18/14: Probes or electrodes therefor
- A61B 2017/00119: {alarm; indicating an abnormal situation}
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2017/00707: {Dummies, phantoms; Devices simulating patient or parts of patient}
- A61B 2017/00716: {simulating physical properties}
- A61B 2018/00297: {Means for providing haptic feedback}
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G06T 2207/10068: Endoscopic image
- G06T 2207/10072: Tomographic images
- G06T 2207/30004: Biomedical image processing
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B33Y: ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

An apparatus has a device representing an endoscope, the device being either an endoscope or a dummy endoscope having shape and feel resembling an endoscope, and includes a tracker adapted to operate with a three-dimensional tracking system to track location and orientation of the device in three dimensions in a simulated operating-room environment. The apparatus also has a physical head model comprising hard and soft components, the device representing an endoscope configured to be inserted into the physical head model to provide a haptic feedback of endoscopic surgery.

## Claims

### Claim 1

A method of preparing a physical model of a head and endoscope for surgical simulation comprising:
importing into a workstation a radiological tomographic image stack of a head;
segmenting the radiological tomographic image stack into hard tissue, soft tissue, and mucosal voxel models based at least in part on voxel intensity;
growing hard tissue, mucosal, and soft tissue regions in the hard tissue, soft tissue, and mucosal voxel models;
converting the hard tissue, soft tissue, and mucosal models into a hard tissue mesh model, a soft tissue mesh model, and a mucosal mesh model;
repairing the mesh models;
exporting the mesh models from the workstation and using at least a 3D printer and the hard tissue mesh model to generate a physical head model;
loading the mesh models into a display system adapted to render images of surfaces of the mesh models as viewed from an endoscope;
mounting a tracker to the physical head model; and
preparing an endoscope device with a tracker.

### Claim 2 (depends on 1)

The method of claim 1 wherein generating a physical head model is performed by printing a hard plastic physical hard tissue model from the hard tissue mesh model;
preparing a resilient polymer physical mucosal tissue model from the mucosal mesh model;
mounting the physical mucosal tissue model to the physical hard tissue model;
preparing a resilient polymer physical soft tissue model from the soft tissue mesh model; and
mounting the physical soft tissue model to the physical hard tissue model to form the physical head model.

### Claim 3 (depends on 1)

The method of claim 1 further comprising:
tracking the endoscope device to determine a tracked endoscope position and orientation;
determining a location and orientation of a tip of the endoscope device from the tracked endoscope position and orientation, the position of the tip of the endoscope device being within the physical head model;
rendering images of surfaces of the mesh models as viewed from the tip of the endoscope device;
displaying the images of surfaces of the mesh models.

### Claim 4 (depends on 3)

The method of claim 3 further comprising:
mounting a tracker to a device representing a surgical tool;
tracking the device representing a surgical tool to determine a location of a tip of the device representing a surgical tool;
determining when the surgical tool is in view of the tip of the endoscope device; and
when the surgical tool is in view of the tip of the endoscope device, rendering an image of a surgical tool as viewed from the tip of the endoscope device.

### Claim 5 (depends on 4)

The method of claim 4 further comprising:
tracking location and orientation of the physical head model; and
registering the mucosal mesh model to the location and orientation of the physical head model.

### Claim 6 (depends on 1)

The method of claim 1 where the rendering images of surfaces of the mesh models is performed with a 3D gaming engine.

### Claim 7 (depends on 2)

The method of claim 2 wherein the preparing a physical mucosal model is performed by casting using a mold that has been prepared from the mucosal mesh model by a method comprising 3D printing.

### Claim 8 (depends on 1, 2, 3, 4, 5, 6)

The method of claim 1, 2, 3, 4, 5, 6, or 7 further comprising identifying critical anatomic structures imaged in the radiological tomographic image stack and tagging those critical structures in a model of the mesh models.

### Claim 9 (depends on 8)

The method of claim 8 further comprising generating alarms upon approach of a tip of the device representing a surgical tool to a critical structure tagged in the mesh model.

### Claim 10 (depends on 9)

The method of claim 9 further comprising generating a score based upon at least time of a trainee completing a task and approach of simulated endoscope tip and simulated tool tip to the critical structures tagged in the model.

### Claim 11 (depends on 8)

The method of claim 8 wherein the physical head model and the endoscope device are configured to physically interact upon insertion of the endoscope device into a nasal cavity of the physical head model and thereby providing haptic feedback to a user, the haptic feedback approximating haptic feedback when the user inserts a real endoscope into a nasal cavity of a real human head.