# US 2024/0252250 A1 · Portable virtual endoscopic tool simulator with haptic feedback enhanced handheld controller

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0252250 A1
- Document ID: US20240252250A1
- Status: Published application
- Kind: A1
- Application number: 18/563,955
- Filed: 2022-05-18
- Published: 2024-08-01
- Earliest claimed: 2021-05-26
- Inventors: Zhan Tao WANG; Jeremy J. CEPEK
- Web page: https://patentgrep.com/patent/US20240252250A1

## CPC

- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- G09B 23/285: {for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas}
- A61B 1/00066: {Proximal part of endoscope body, e.g. handles (A61B1/0052 takes precedence)}
- A61B 1/307: for the urinary organs, e.g. urethroscopes or cystoscopes
- A61B 34/25: {User interfaces for surgical systems}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/2057: {Details of tracking cameras}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An augmented reality/virtual reality (AR/VR) system, including a handheld controller, at least two imaging devices and a computer, is capable of simulating a variety of endoscopic, especially urological, procedures, for the purpose of training and/or equipment design evaluation. The handheld controller has inertial sensors for three axes of motion, user actuatable controls for causing the virtual urological endoscope to interact with a virtual urological environment and a haptic feedback generator for providing haptic feedback to the user. The at least two imaging devices are configured to obtain position information relating to the handheld controller and to provide the position information to the computer.

## Claims

### Claim 1

A urological procedure simulator comprising:
a user interface for control of a virtual urological endoscope by a user; and,
a computer in electronic communication with the user interface, the computer configured with computer executable instructions to display the virtual urological endoscope and a virtual urological environment on a display screen, the computer further configured with computer executable instructions to adjust a position of the virtual urological endoscope on the display screen in response to user commands received from the user interface,
the user interface comprising
a handheld controller comprising inertial sensors for three axes of motion, user actuatable controls for causing the virtual urological endoscope to interact with the virtual urological environment, and a haptic feedback generator for providing haptic feedback to the user, and
at least two imaging devices configured to obtain position information relating to the handheld controller and to provide the position information to the computer.

### Claim 2 (depends on 1)

The simulator of claim 1, wherein the user interface and the computer are configured with wireless communication interfaces for wireless electronic communication and the computer is in wireless electronic communication with the user interface.

### Claim 3 (depends on 1)

The simulator of claim 1, wherein the user interface further comprises an optical unit positionable remotely from the handheld controller, wherein the optical unit comprises the at least two imaging devices and each imaging device has a field of view that includes the handheld controller.

### Claim 4 (depends on 1)

The simulator of claim 1, wherein the user interface further comprises an optical unit positionable remotely from the handheld controller, wherein the at least two imaging devices are situated in the controller and each imaging device has a field of view that includes the optical unit.

### Claim 5 (depends on 1)

The simulator of claim 1, wherein the handheld controller is in electronic communication with the computer.

### Claim 6 (depends on 1)

The simulator of claim 1, wherein the computer is configured with computer executable instructions to track user performance metrics relating to interaction between the virtual urologic endoscope and the virtual urological environment and to provide feedback to the user relating to the performance metrics during a simulated urological procedure and/or after completion of the simulated urological procedure.

### Claim 7 (depends on 1)

The simulator of claim 1, wherein the inertial sensors sense translation along and/or rotation about the three axes of motion of the handheld controller.

### Claim 8 (depends on 1)

The simulator of claim 1, wherein the virtual urological environment contains a virtual target, the virtual urological target comprising a kidney stone, a lesion on a wall of a urethra, a bladder tumor or a portion of the virtual urological environment from which to take a biopsy sample.

### Claim 9 (depends on 1)

The simulator of claim 1, wherein the virtual urological endoscope further comprises a virtual endoscopic tool.

### Claim 10 (depends on 9)

The simulator of claim 9, wherein the virtual endoscopic tool comprises a wire, an electrode, a laser fiber, an ultrasound device, a grasper, biopsy forceps, a scalpel, a stone basket, a loop or a roller-ball.

### Claim 11 (depends on 1)

The simulator of claim 1, wherein the user actuatable controls comprise a user actuatable control for curling the tip of the virtual urological endoscope.

### Claim 12 (depends on 1)

The simulator of claim 1, wherein the user actuatable controls for causing the virtual urological endoscope to interact with the virtual urological environment comprise buttons and/or knobs.

### Claim 13 (depends on 1)

The simulator of claim 1, wherein the at least two imaging devices comprise optical cameras.

### Claim 14 (depends on 1)

The simulator of claim 1, wherein the computer is a mobile device, a laptop computer or a desktop computer.

### Claim 15 (depends on 14)

The simulator of claim 14, wherein the mobile device is a tablet computer or a portable telephone.
