# US 10,478,255 B2 · System and method for holographic image-guided percutaneous endovascular percutaneous procedures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,478,255 B2 - Document ID: US10478255B2 - Status: Granted - Kind: B2 - Application number: 15/958,770 - Filed: 2018-04-20 - Published: 2018-10-25 - Granted: 2019-11-19 - Earliest claimed: 2017-04-20 - Assignees: The Cleveland Clinic Foundation - Applicants: The Cleveland Clinic Foundation - Inventors: Karl West; Jeffrey H. Yanof - Web page: https://patentgrep.com/patent/US10478255B2 ## CPC - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 5/062: {using magnetic field} - A61B 5/6852: {Catheters} - A61B 5/7405: {using sound} - A61B 5/745: {using a holographic display} - A61B 34/25: {User interfaces for surgical systems} - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 90/37: {Surgical systems with images on a monitor during operation} - A61M 5/00: Devices for bringing media into the body in a subcutaneous, intra-vascular or intramuscular way; Accessories therefor, e.g. filling or cleaning devices, arm rests (tube connectors, tube couplings, valves or branch units, specially adapted for medical use A61M39/00; containers specially adapted for medical or pharmaceutical purposes A61J1/00) - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038) - G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning - G06T 7/344: {involving models} - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06V 20/20: in augmented reality scenes - A61B 5/064: {using markers (A61B5/062 takes precedence)} - A61B 2034/102: {Modelling of surgical devices, implants or prosthesis} - A61B 2034/107: {Visualisation of planned trajectories or target regions} - A61B 2034/2051: {Electromagnetic tracking systems} - A61B 2034/2057: {Details of tracking cameras} - A61B 2034/2068: {using pointers, e.g. pointers having reference marks for determining coordinates of body points} - A61B 2090/363: {Use of fiducial points} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/366: {using projection of images directly onto the body} - A61B 2090/367: {creating a 3D dataset from 2D images using position information} - A61B 2090/368: {changing the image on a display according to the operator's position} - A61B 2090/372: {Details of monitor hardware} - A61B 2090/3762: {using computed tomography systems [CT]} - A61B 2090/3983: {Reference marker arrangements for use with image guided surgery} - A61B 2090/502: {Headgear, e.g. helmet, spectacles} - A61M 2205/507: Head Mounted Displays [HMD] - G02B 2027/0174: {holographic} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 10/245: {by locating a pattern; Special marks for positioning} - G06V 2201/03: Recognition of patterns in medical or anatomical images - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - A61M: DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY (introducing media into or onto the bodies of animals A61D7/00; means for inserting tampons A61F13/26; devices for administering food or medicines orally A61J; containers for collecting, storing or administering blood or medical fluids A61J1/05); DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY (surgery A61B; chemical aspects of surgical articles A61L); DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR {(Electrotherapy, e.g. producing anaesthesia by the use of alternating or intermittent currents A61N1/36021)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Holographic image-guidance can be used to track an interventional device during an endovascular percutaneous procedure. The holographic image guidance can be provided by a head-mounted device by transforming tracking data and vasculature image data to a common coordinate system and creating a holographic display relative to a patient's vasculature to track the interventional device during the endovascular percutaneous procedure. The holographic display can also include graphics to provide guidance for the physical interventional device as it travels through the patient's anatomy (e.g., the vasculature). ## Claims ### Claim 1 A method comprising: receiving, by a head-mounted device comprising a processor, tracking data for a physical interventional device in a tracking coordinate system, wherein the physical interventional device is used during an endovascular percutaneous medical procedure; transforming, by the head-mounted device, the tracking data for the physical interventional device in the tracking coordinate system into a headset coordinate system; accessing, by the head-mounted device, image data from a pre-operative image of a patient's anatomy comprising a physical operative site in an imaging coordinate system; transforming, by the head mounted device, the image data in the imaging coordinate system into the headset coordinate system; registering, by the head-mounted device, a 3D holographic representation of the interventional device based on the tracking data for the physical interventional device in the headset coordinate system to 3D anatomical holographic projections of the patient's anatomy based on the imaging data in the headset coordinate system; displaying, by the head mounted device, the 3D anatomical holographic projections providing a visualization of a holographic version of the patient's anatomy with reference graphics related to a physical operative site within the patient's anatomy; displaying, by the head mounted device, the 3D holographic representation of the interventional device providing a visualization of a holographic version of the interventional device with guidance control graphics related to the physical interventional device; and navigating, by the head mounted device, the 3D holographic representation of the interventional device in the 3D anatomical holographic projections based on the tracking data for the interventional device in the headset coordinate system, wherein the reference graphics and the guidance control graphics provide guidance for tracking the physical interventional device through the patient's anatomy using the 3D anatomical holographic projections and the 3D holographic representation of the interventional device. ### Claim 2 (depends on 1) The method of claim 1, wherein the physical interventional device comprises an integrated sensor-coil that is detectable by a tracking system to provide the tracking data in the tracking coordinate system. ### Claim 3 (depends on 2) The method of claim 2, wherein the tracking data has six-degrees of freedom (6 DOF). ### Claim 4 (depends on 2) The method of claim 2, wherein the tracking data comprises real-time 3D position data and real-time 3D orientation data. ### Claim 5 (depends on 2) The method of claim 2, wherein the sensor coil is located at a tip of the physical interventional device. ### Claim 6 (depends on 1) The method of claim 1, wherein the image data from the pre-operative image of a patient's anatomy is represented by patient-specific DICOM image data from one or more 3D pre-operative tomographic image data sets. ### Claim 7 (depends on 1) The method of claim 1, wherein the 3D anatomical holographic projections are based on one or more surface mesh models or multi-planar reformatted images created from the image data. ### Claim 8 (depends on 1) The method of claim 1, further comprising transforming, by the head mounted device, the visualization by at least one of translating, rotating, and scaling to enhance the navigating. ### Claim 9 (depends on 8) The method of claim 8, wherein the transforming is triggered by a physical movement of the head mounted device. ### Claim 10 (depends on 1) The method of claim 1, wherein the transforming further comprises: locating positions of three or more fiducial markers on the physical patient's anatomy, wherein the positions are in the tracking coordinate system; and matching the positions of the three or more fiducial markers to three or more locations in the image data in the image coordinate system. ### Claim 11 (depends on 10) The method of claim 10, wherein the positions of the three or more fiducial markers are non-collinear. ### Claim 12 (depends on 10) The method of claim 10, wherein the three or more fiducial markers are sensor-coils placed on the patient's skin. ### Claim 13 (depends on 1) The method of claim 1, wherein the endovascular percutaneous medical procedure comprises an aneurism repair procedure, a stent grafting/placement procedure, a catheterization procedure, a filter placement procedure, an angioplasty procedure, a minimally-invasive catheter procedure, an ablation procedure, a cryoablation procedure, or a guidewire placement procedure. ### Claim 14 (depends on 1) The method of claim 1, wherein the guidance comprises visual feedback or auditory feedback related to location and orientation of the physical interventional device. ### Claim 15 (depends on 14) The method of claim 14, further comprising providing the visual feedback to monitor when a graphic corresponding to the guidance control graphics intersects a graphic corresponding to the reference graphics indicating that the physical interventional device is in alignment with a trajectory that would place the physical interventional device at a target within the patient's vasculature. ### Claim 16 (depends on 15) The method of claim 15, wherein the visual feedback includes a holographic annotation that reports the distance or angle deviation from a targeted position or orientation. ### Claim 17 A head-mounted device comprising: a non-transitory memory storing instructions; and a processor to access the non-transitory memory and execute the instructions to: receive tracking data for a physical interventional device in a tracking coordinate system, wherein the physical interventional device is used during an endovascular percutaneous medical procedure; transform the tracking data for the physical interventional device in the tracking coordinate system into a headset coordinate system; access image data from a pre-operative image of a patient's anatomy comprising a physical operative site in an imaging coordinate system; transform the image data in the imaging coordinate system into the headset coordinate system; register a 3D holographic representation of the interventional device based on the tracking data for the physical interventional device in the headset coordinate system to 3D anatomical holographic projections of the patient's anatomy based on the imaging data in the headset coordinate system; display the 3D anatomical holographic projections providing a visualization of a holographic version of the patient's anatomy with reference graphics related to a physical operative site within the patient's anatomy; displaying, by the head mounted device, the 3D holographic representation of the interventional device providing a visualization of a holographic version of the interventional device with guidance control graphics related to the physical interventional device; and navigate the 3D holographic representation of the interventional device in the 3D anatomical holographic projections based on the tracking data for the interventional device in the headset coordinate system, wherein the reference graphics and the guidance control graphics provide guidance for tracking the physical interventional device through the patient's anatomy using the 3D anatomical holographic projections and the 3D holographic representation of the interventional device. ### Claim 18 (depends on 17) The head-mounted device of claim 17, further comprising a head mounted display to display the visualization. ### Claim 19 (depends on 18) The head-mounted device of claim 18, wherein the processor further executes the instructions to transform the visualization by at least one of translating, rotating, and scaling to enhance the navigating triggered by a physical movement of the head mounted device. ### Claim 20 (depends on 17) The head-mounted device of claim 17, wherein the transforming further comprises: locating positions of three or more fiducial markers on the physical patient's skin, wherein the positions are in the tracking coordinate system, wherein the positions are non-collinear; and matching the positions of the three or more fiducial markers to three or more locations in the image data in the image coordinate system.