# US 10,869,727 B2 · Live 3D holographic guidance and navigation for performing interventional procedures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,869,727 B2 - Document ID: US10869727B2 - Status: Granted - Kind: B2 - Application number: 16/405,875 - Filed: 2019-05-07 - Published: 2019-11-07 - Granted: 2020-12-22 - Earliest claimed: 2018-05-07 - Assignees: The Cleveland Clinic Foundation - Applicants: The Cleveland Clinic Foundation - Inventors: Jeffrey H. Yanof; Karl West; Sara Al-Nimer - Web page: https://patentgrep.com/patent/US10869727B2 ## CPC - A61B 8/462: {characterised by constructional features of the display} - A61B 8/463: {characterised by displaying multiple images or images and diagnostic data on one display} - A61B 8/466: {adapted to display 3D data} - A61B 8/483: {involving the acquisition of a 3D volume of data} - A61B 8/5238: {for combining image data of patient, e.g. merging several images from different acquisition modes into one image} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06V 20/20: in augmented reality scenes - A61B 8/4245: {involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient} - A61B 8/465: {adapted to display user selection data, e.g. icons or menus} - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2034/107: {Visualisation of planned trajectories or target regions} - A61B 2034/2048: {using an accelerometer or inertia sensor} - A61B 2034/2051: {Electromagnetic tracking systems} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2063: {Acoustic tracking systems, e.g. using ultrasound} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/368: {changing the image on a display according to the operator's position} - A61B 2090/372: {Details of monitor hardware} - A61B 2090/378: {using ultrasound} - A61B 2090/502: {Headgear, e.g. helmet, spectacles} - A61M 2205/507: Head Mounted Displays [HMD] - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G06F 3/012: {Head tracking input arrangements} - G06V 10/245: {by locating a pattern; Special marks for positioning} - G06V 2201/03: Recognition of patterns in medical or anatomical images - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61M: DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY (introducing media into or onto the bodies of animals A61D7/00; means for inserting tampons A61F13/26; devices for administering food or medicines orally A61J; containers for collecting, storing or administering blood or medical fluids A61J1/05); DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY (surgery A61B; chemical aspects of surgical articles A61L); DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR {(Electrotherapy, e.g. producing anaesthesia by the use of alternating or intermittent currents A61N1/36021)} ## Abstract An interventional procedure can be performed less invasively with live 3D holographic guidance and navigation. Advantageously, the 3D holographic guidance and navigation overcomes the limitations of conventional displays of medical images on standard 2D flat panel screens. The live 3D holographic guidance can utilize a 3D holographic visualization that is derived from complementary imaging modalities (e.g., ultrasound and computed tomography) to provide a complete holographic view of a portion of the patient's body to enable navigation of a tracked interventional instrument/device. ## Claims ### Claim 1 A method comprising: receiving, by a physical head-mounted device comprising a processor and a head-tracking mechanism, live tracking data from a procedure in 3D Cartesian coordinates of a navigation system, wherein the live tracking data comprises a position and orientation of: a tracked physical ultrasound transducer/probe connected to an ultrasound system, a tracked physical interventional device/instrument, and physical fiducial location sensors at specific anatomical locations on a physical patient; receiving, by the physical head-mounted device, a live ultrasound image stream acquired by the physical ultrasound transducer/probe connected to the ultrasound system; transforming, by the head-mounted device, the live ultrasound image stream to a headset coordinate system; transforming, by the head-mounted device, the live tracking data to the headset coordinate system; displaying, in a head-mounted display of the head mounted device, a live holographic projection of the live ultrasound image stream in the headset coordinate system, wherein the live holographic projection of the ultrasound image stream is based on the tracked position and tracked orientation of the transducer/probe, wherein the live holographic projection is scaled to the physical anatomy consistent with an operator's view assessed with the head-tracking mechanism; retrieving, by the head mounted device, digital anatomical objects derived from pre-operative 3D computed tomography (CT) image data of the physical patient, wherein the CT image data is in 3D coordinates of a CT coordinate system; transforming, by the head-mounted device, the digital anatomical objects from the 3D CT coordinate system to the headset coordinate system; translating, by the head mounted device, the anatomical objects in the headset coordinate system by a 3D vector computed based on a 3D point location on the live holographic projection of the live image stream and a corresponding point within the stored pre-operative CT image data in the headset coordinate system to correct for a live anatomical motion of the physical patient during the procedure; and displaying, in the head mounted display of the head mounted device, a holographic visualization comprising a holographic representation of the tracked physical interventional device/instrument congruent with the registered holographic projection of the live ultrasound image stream and the holographic anatomical objects derived from CT, wherein the holographic visualization is used to navigate the tracked physical ultrasound transducer/probe and guide the physical interventional device/instrument to a therapeutic target during the procedure. ### Claim 2 (depends on 1) The method of claim 1, wherein the live anatomical motion comprises respiratory motion or gross patient motion. ### Claim 3 (depends on 1) The method of claim 1, wherein the displaying further comprises displaying a holographic representation of the interventional device/instrument with guidance control graphics within the 3D anatomical holographic projections comprising reference graphics related to an operative site comprising at least a portion of the portion of the patient's physical anatomy, wherein the guidance control graphics and the reference graphics are configured to aid in navigating the interventional device/instrument through the portion of the patient's physical anatomy to the therapeutic point. ### Claim 4 (depends on 1) The method of claim 1, wherein the physical interventional device/instrument is utilized in a percutaneous non-vascular medical procedure, an endovascular percutaneous medical procedure, or an open surgical procedure. ### Claim 5 (depends on 1) The method of claim 1, wherein the holographic anatomical objects derived from CT are projected to enable visualization without occlusion of the live holographic projection of the live ultrasound image stream. ### Claim 6 (depends on 1) The method of claim 1, wherein a 3D point is identified and located on the registered holographic projection of the live ultrasound image stream based on an operator input, wherein the 3D point is a center of an imaged tumor or blood vessel cross section. ### Claim 7 (depends on 1) The method of claim 1, wherein the live tracking data is based on electromagnetic tracking of electromagnetic sensor coils or optical tracking of reflective markers. ### Claim 8 (depends on 7) The method of claim 7, wherein the electromagnetic sensor coils and/or the reflective markers are located within and/or on the tracked physical ultrasound transducer/probe, the tracked physical interventional device/instrument, and the physical fiducial location sensors. ### Claim 9 (depends on 1) The method of claim 1, wherein the holographic visualization is stereoscopically projected onto the patient in congruence with the patient's physical anatomy, the ultrasound transducer/probe, and the physical interventional device/instrument. ### Claim 10 (depends on 9) The method of claim 9, further comprising, upon receiving an operator input, projecting, by the head mounted device, the visualization not congruent with the patient's physical anatomy and/or the physical ultrasound transducer/probe and/or the physical interventional device/instrument while maintaining co-registration of the holographic projections. ### Claim 11 (depends on 9) The method of claim 9, further comprising, upon receiving an operator input, continuously realigning, by the head mounted device, the visualization to re-orient the virtual display so that it always faces the operator based on tracking data from the head mounted device. ### Claim 12 (depends on 9) The method of claim 9, further comprising maintaining a co-registration of a holographic light ray, the registered holographic projection of the live ultrasound image stream, and the holographic anatomical objects derived from CT when the visualization is translated, rotated, and/or scaled relative to the physical anatomy. ### Claim 13 (depends on 12) The method of claim 12 wherein, the intersection of the holographic light ray and the registered holographic projection of the live ultrasound image stream is displayed as a spherical point of intersection. ### Claim 14 (depends on 9) The method of claim 9, further comprising, by limiting or preventing the rotation of the holographic scene to maintain hand-eye coordination of the operator when navigating the ultrasound transducer/probe and the interventional device/instrument. ### Claim 15 (depends on 1) The method of claim 1, wherein the registered holographic projection of the live ultrasound image stream is pre-calibrated with local rotation, translation, and scaling to extend from the tracked physical interventional device/instrument in congruence with the patient's physical anatomy. ### Claim 16 (depends on 15) The method of claim 15, wherein the registered holographic projection of the live ultrasound image stream is automatically re-calibrated with the local rotation, translation, and scaling so that it maintains congruence with the patient's physical anatomy when the operator adjusts a scale/depth setting on the physical ultrasound system. ### Claim 17 A system comprising: a computing device comprising a memory to store digital anatomical objects derived from pre-operative 3D computed tomography (CT) image data of a physical patient, wherein the CT image data is in 3D coordinates of a CT coordinate system; a head-mounted device, comprising a processor, a head mounted display, and a head-tracking mechanism, to: receive live tracking data from a procedure in 3D Cartesian coordinates of a navigation system, wherein the live tracking data comprises a position and orientation of: a tracked physical ultrasound transducer/probe connected to an ultrasound system, a tracked physical interventional device/instrument, and physical fiducial location sensors at specific anatomical locations on a physical patient; receive a live ultrasound image stream acquired by the physical ultrasound transducer/probe connected to the ultrasound system; transform the live ultrasound image stream to a headset coordinate system; transform the live tracking data to the headset coordinate system; display, in the head-mounted display, a live holographic projection of the live ultrasound image stream in the headset coordinate system, wherein the live holographic projection of the ultrasound image stream is based on the tracked position and tracked orientation of the transducer/probe, wherein the live holographic projection is scaled consistent with an operator's view assessed with the head-tracking mechanism; retrieve the digital anatomical objects; transform the digital anatomical objects from the 3D coordinates of the CT coordinate system to the headset coordinate system; translate the anatomical objects in the headset coordinate system by a 3D vector computed based on a 3D point location on the live holographic projection of the live image stream and a corresponding point within the stored pre-operative CT image data in the headset coordinate system to correct for a live anatomical motion of the physical patient during the procedure; and display, in the head mounted display, a holographic visualization comprising a holographic representation of the tracked physical interventional device/instrument congruent with the registered holographic projection of the live ultrasound image stream and the holographic anatomical objects derived from CT, wherein the holographic visualization is used to position the tracked physical ultrasound transducer/probe and guide the physical interventional device/instrument to a therapeutic target. ### Claim 18 (depends on 17) The system of claim 17, wherein the live tracking data is based on electromagnetic tracking of electromagnetic sensor coils and/or optical tracking of reflective markers, and wherein the electromagnetic sensor coils and/or the reflective markers are located within and/or on the tracked physical ultrasound transducer/probe, the tracked physical interventional device/instrument, and the physical fiducial location sensors. ### Claim 19 (depends on 17) The system of claim 17, wherein the holographic visualization is stereoscopically projected onto the patient in congruence with the patient's physical anatomy, the ultrasound transducer/probe, and the physical interventional device/instrument. ### Claim 20 (depends on 19) The system of claim 19, further comprising, upon receiving an operator input, projecting, by the head mounted device, the visualization not congruent with the patient's physical anatomy and/or the physical ultrasound transducer/probe and/or the physical interventional device/instrument while maintaining co-registration of the holographic projections. ### Claim 21 (depends on 19) The system of claim 19, further comprising, upon receiving an operator input, continuously realigning, by the head mounted device, the visualization to always face the operator based on tracking data from the head mounted device. ### Claim 22 (depends on 19) The system of claim 19, further comprising maintaining a co-registration of a holographic light ray, the registered holographic projection of the live ultrasound image stream, and the holographic anatomical objects derived from CT when the visualization is translated, rotated, and/or scaled relative to the physical anatomy. ### Claim 23 (depends on 22) The system of claim 22 wherein, the intersection of the holographic light ray and the registered holographic projection of the live ultrasound image stream is displayed as a spherical point of intersection.