# US 12,118,895 B2 · Mixed-reality endoscope and surgical tools with haptic feedback for integrated virtual-reality visual and haptic surgical simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,895 B2
- Document ID: US12118895B2
- Status: Granted
- Kind: B2
- Application number: 17/048,991
- Filed: 2019-04-18
- Published: 2021-08-12
- Granted: 2024-10-15
- Earliest claimed: 2018-04-18
- Assignees: Arizona Board of Regents on Behalf of the University of Arizona
- Applicants: Arizona Board of Regents On Behalf of the University of Arizona
- Inventors: Samuel Barber; Saurabh Jain; Young-Jun Son; Eugene Chang
- Web page: https://patentgrep.com/patent/US12118895B2

## CPC

- G09B 23/285: {for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas}
- A61B 17/24: for use in the oral cavity, larynx, bronchial passages or nose (for medical inspection of cavities or tubes in the body A61B1/00); Tongue scrapers
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- B33Y 80/00: Products made by additive manufacturing
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/11: Region-based segmentation
- G09B 23/30: Anatomical models {(G09B23/281 - G09B23/288 take precedence)}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 40/63: for local operation
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 17/29: Forceps for use in minimally invasive surgery
- A61B 17/320016: {Endoscopic cutting instruments, e.g. arthroscopes, resectoscopes (A61B10/0233, A61B17/29, A61B17/3207 take precedence)}
- A61B 17/3201: Scissors (for biopsy A61B10/02)
- A61B 18/14: Probes or electrodes therefor
- A61B 2017/00119: {alarm; indicating an abnormal situation}
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2017/00707: {Dummies, phantoms; Devices simulating patient or parts of patient}
- A61B 2017/00716: {simulating physical properties}
- A61B 2018/00297: {Means for providing haptic feedback}
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G06T 2207/10068: Endoscopic image
- G06T 2207/10072: Tomographic images
- G06T 2207/30004: Biomedical image processing
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B33Y: ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

An apparatus has a device representing an endoscope, the device being either an endoscope or a dummy endoscope having shape and feel resembling an endoscope, and includes a tracker adapted to operate with a three-dimensional tracking system to track location and orientation of the device in three dimensions in a simulated operating-room environment. The apparatus also has a physical head model comprising hard and soft components, the device representing an endoscope configured to be inserted into the physical head model to provide a haptic feedback of endoscopic surgery.

## Claims

### Claim 1

A multimode VR apparatus comprising:
an endoscope device selected from an endoscope and a dummy endoscope having shape and feel resembling that of an endoscope,
a first wireless tracker operating with a three-dimensional tracking system to track location and orientation of the endoscope device in three dimensions in a simulated operating-room environment,
a video modeling and display machine loaded with a computer-aided design (CAD) model of a head, providing a simulated endoscope view of a simulated head environment; and
a physical head model comprising hard and soft physical components, and a second wireless tracker where the computer-aided design (CAD) model of the head is registered to a tracked position of the physical head model, the endoscope device being configured to be inserted into the physical head model to provide haptic feedback during manipulation resembling haptic feedback present during manipulation of an endoscope in a head to a person handling the endoscope device.

### Claim 2 (depends on 1)

The apparatus of claim 1 wherein the video modeling and display machine comprises a gaming engine simulating an endoscope view of the simulated head environment.

### Claim 3 (depends on 2)

The apparatus of claim 2 wherein the physical head model comprises a hard-plastic portion prepared by 3D printing representative of bony tissue and a resilient polymer portion representative of skin.

### Claim 4 (depends on 1)

The apparatus of claim 1 further comprising a surgical tool device having shape and feel resembling that of a surgical tool adapted for functional endoscopic sinus surgery (FESS), the surgical tool device further comprising a wireless tracker adapted to operate with the three-dimensional tracking system to track location and orientation of the surgical tool device in three dimensions in the simulated head environment.

### Claim 5 (depends on 4)

The apparatus of claim 4 wherein the simulated endoscope view of the simulated head environment includes a simulated endoscope view of the surgical tool device.

### Claim 6 (depends on 5)

The apparatus of claim 5 wherein the surgical tool device resembles a surgical tool selected from the group consisting of forceps, biting forceps, scissors, a probe, a drill, and an electrocautery.

### Claim 7 (depends on 6)

The apparatus of claim 6 wherein the video modeling and display machine is further configured to provide a simulated operating room (OR) environment with the simulated endoscope view displayed on a simulated endoscope monitor.

### Claim 8 (depends on 7)

The apparatus of claim 7 further comprising a virtual-reality (VR) goggle equipped with a wireless tracker, and wherein the simulated OR environment is displayed through the VR goggle with position and size of the simulated endoscope monitor on the VR goggle display dependent on angle and position of the VR goggle.