# US 2018/0322806 A1 · Hybrid image/scene renderer with hands free control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322806 A1
- Document ID: US20180322806A1
- Status: Published application
- Kind: A1
- Application number: 16/033,937
- Filed: 2018-07-12
- Published: 2018-11-08
- Earliest claimed: 2012-05-25
- Assignees: Surgical Theater, Inc.
- Applicants: Surgical Theater LLC
- Inventors: Mordechai Avisar; Alon Yakob Geri
- Web page: https://patentgrep.com/patent/US20180322806A1

## CPC

- G09B 23/28: for medicine
- A61B 17/1227: {Spring clips}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 17/12: for ligaturing or otherwise compressing tubular parts of the body, e.g. blood vessels or umbilical cord
- A61B 17/1285: {for minimally invasive surgery}
- A61B 2017/00707: {Dummies, phantoms; Devices simulating patient or parts of patient}
- A61B 2034/104: {Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A system and method for converting static/still medical images of a particular patient into dynamic and interactive images interacting with medical tools including medical devices by coupling a model of tissue dynamics and tool characteristics to the patient specific imagery for simulating a medical procedure in an accurate and dynamic manner by coupling a model of tissue dynamics to patient specific imagery for simulating surgery on the particular patient. The method includes a tool to add and/or to adjust the dynamic image of tissues and ability to draw any geometric shape on the dynamic image of tissues and to add the shape into the modeling system.

## Claims

### Claim 1

A modeling system for enabling a user to perform a simulated medical procedure, said system comprising:
one or more computers;
a display for displaying images to the user;
a database for storing characteristics of the tissues of a particular patient;
an image generator using one or more of said computers for executing software for generating a dynamic realistic image of the tissues of the particular patient based on the stored characteristics of the particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance including shadowing and textures indicative of the tissues of the particular patient;
a database for storing a user tool library for providing a plurality of user tool models of actual user tools used in medical procedures;
a user interface for accepting inputs from the user for selecting one of the user tool models;
a user tool generator using one or more of said computers for executing software for generating a realistic tool image of the selected user tool model for displaying on said display; and
a user interface including a camera for accepting inputs from the user based on motions of the user's hands, said inputs for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time, wherein
the dynamic interaction between the user tool image and the image of the tissues is displayed on said display using images with realistic visual features exhibiting realistic mechanical interactions.

### Claim 2 (depends on 1)

The modeling system of claim 1, further comprising a tool to adjust the dynamic image of tissues to provide the ability to draw any geometric shape on the dynamic image of tissues.

### Claim 3 (depends on 2)

The modeling system of claim 2, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to complete an incomplete anatomical structure of the dynamic image of tissues.

### Claim 4 (depends on 1)

The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues provides the ability to modify the texture, lighting, shadow and/or shading of a portion of the dynamic image of tissues

### Claim 5 (depends on 1)

The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to command the tool to interact with one or more portions of the dynamic image of tissues.

### Claim 6 (depends on 1)

The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to select elements of a model of the tool and/or the dynamic image of tissues for removal from the displayed image.

### Claim 7 (depends on 1)

The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to reposition objects in the displayed image by selecting the objects and dragging the objects to a desired position for display in the image.

### Claim 8 (depends on 1)

The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to enhance and integrate anatomical structure in the dynamic image.

### Claim 9 (depends on 1)

The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to a draw any geometric shape for adding to the dynamic image of tissues.

### Claim 10 (depends on 1)

The modeling system of claim 1, further comprising:
a database for storing a library of a plurality of models of different implants; and
a user interface for selecting one implant model from said plurality of models for use with said user tool model for dynamically interacting with said image of tissues.

### Claim 11 (depends on 1)

The modeling system of claim 1, wherein a tool to adjust the dynamic image of tissues includes tools to provide the abilities to:
draw any geometric shape on the dynamic image of tissues;
complete an incomplete anatomical structure of the dynamic image of tissues; or
modify the texture, and/or lighting of a portion of the dynamic image of tissues.

### Claim 12 (depends on 1)

The modeling system of claim 1, wherein the inputs from the user based on motions of the user's hands are configured to track the motions of an actual surgical instrument being used by the user with the particular patient, such that said motions are used for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time.

### Claim 13 (depends on 12)

The modeling system of claim 12, wherein said user interface that can track the motions of an actual surgical instrument includes a GPS receiver, an accelerometer, a magnetic detection device, or a camera.

### Claim 14

A modeling system for enabling a user to perform a simulated medical procedure, said system comprising:
one or more computers;
a display for displaying images to the user;
a database for storing physical characteristics of the tissues of a particular patient;
an image generator using one or more of said computers for executing software for generating a dynamic realistic image of the tissues of the particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance including shadowing and textures indicative of the tissues of the particular patient;
a database comprising a user tool library for providing a plurality of user tool models of actual user tools used in medical procedures;
a user interface for accepting inputs from the user for selecting one of the user tool models;
a user tool generator using one or more of said computers for executing software for generating a realistic tool image of the selected user tool model for displaying on said display;
a user interface for accepting inputs from the user, said inputs for dynamically manipulating said selected user tool image for dynamically interacting with said realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time;
a user interface including a camera for accepting inputs from the user based on motions of the user's hands, said inputs for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time, wherein
the dynamic interaction between the user tool image and the image of the tissues is displayed on said display using images with realistic visual features exhibiting realistic mechanical interactions; and
a user interface providing a tool to adjust the dynamic image of the tissues displayed on said display by adding or modifying features of said tissues for display to compensate for anatomical structures that are in the actual biological tissue of the particular patient but are missing from the dynamic image of tissues originally displayed such that the dynamic image of tissues displayed are subsequently displayed on the display with the added or modified features, wherein
the dynamic interaction between the user tool image and the image of the tissues is displayed on said display providing realistic visual features exhibiting realistic mechanical interactions based on the stored physical characteristics.

### Claim 15 (depends on 14)

The modeling system of claim 14, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to draw any geometric shape on the dynamic image of tissues.

### Claim 16 (depends on 14)

The modeling system of claim 14, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to complete an incomplete anatomical structure of the dynamic image of tissues.

### Claim 17 (depends on 14)

The modeling system of claim 14, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to modify the texture, lighting, shadow and/or shading of a portion of the dynamic image of tissues

### Claim 18 (depends on 14)

The modeling system of claim 14, wherein said medical images of the particular patient include an image of an aneurysm, and wherein said dynamic image includes an image of the aneurysm, and further wherein said user tool includes an aneurysm clip applier for applying an aneurysm clip model for dynamically interacting with the image of tissues.

### Claim 19 (depends on 14)

The modeling system of claim 14, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to command a tool to interact with one or more portions of the dynamic image of tissues.

### Claim 20 (depends on 14)

The modeling system of claim 14, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to select elements the tool model and/or the dynamic image of tissues for removal from the displayed image.

### Claim 21 (depends on 14)

The modeling system of claim 14, wherein the tool to adjust the dynamic image of tissues includes a tool to provide the ability to reposition objects in the displayed image by selecting the objects and dragging the objects to a desired position for display in the image.

### Claim 22 (depends on 14)

The modeling system of claim 14, further comprising:
a database for storing a library of a plurality of models of different implants; and
a user interface for selecting one implant model from said plurality of models for use with said user tool model for dynamically interacting with said image of tissues.

### Claim 23 (depends on 14)

The modeling system of claim 14, wherein a tool to adjust the dynamic image of tissues includes tools to provide the abilities to:
draw any geometric shape on the dynamic image of tissues;
complete an incomplete anatomical structure of the dynamic image of tissues; or
modify the texture, and/or lighting of a portion of the dynamic image of tissues.

### Claim 24 (depends on 14)

The modeling system of claim 14, further comprising a user interface that can track the motions of an actual surgical instrument being used by the user with the particular patient, such that said motions are used for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time.

### Claim 25 (depends on 24)

The modeling system of claim 24, wherein said user interface that can track the motions of an actual surgical instrument includes a GPS receiver, an accelerometer, a magnetic detection device, or a camera.

### Claim 26

A modeling system for enabling a user to perform a simulated medical procedure, said system comprising:
one or more computers;
a display for displaying images to the user;
an image generator using one or more of said computers for executing software for generating a dynamic realistic image of the tissues for particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance including shadowing and textures indicative of actual tissues;
a database for storing a user tool library for providing a plurality of user tool models of actual user tools used in medical procedures;
a user interface for accepting inputs from the user for selecting one of the user tool models;
a user tool generator using one or more of said computers for executing software for generating a realistic tool image of the selected user tool model for displaying on said display; and
a user interface that can track the motions of an actual surgical instrument being used by the user with the particular patient, such that said motions are used for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time.
GPS, accelerometers, magnetic detection, or other location and motion detecting devices.

### Claim 27 (depends on 26)

The modeling system of claim 26, wherein said user interface that can track the motions of an actual surgical instrument includes a GPS receiver, an accelerometer, a magnetic detection device, or a camera.

### Claim 28 (depends on 26)

The modeling system of claim 26, further comprising a user interface including a camera for accepting inputs from the user based on motions of the user's hands, said inputs for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time, wherein the dynamic interaction between the user tool image and the image of the tissues is displayed on said display using images with realistic visual features exhibiting realistic mechanical interactions.
