# US 2017/0280989 A1 · Visualization system for ophthalmic surgery > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0280989 A1 - Document ID: US20170280989A1 - Status: Published application - Kind: A1 - Application number: 15/087,585 - Filed: 2016-03-31 - Published: 2017-10-05 - Earliest claimed: 2016-03-31 - Assignees: Novartis AG - Applicants: Novartis AG - Inventors: Tammo Heeren - Web page: https://patentgrep.com/patent/US20170280989A1 ## CPC - A61B 1/313: for introducing through surgical openings, e.g. laparoscopes - A61B 1/000094: {extracting biological structures} - A61B 1/000095: {for image enhancement} - A61B 1/01: Guiding arrangements therefore - A61B 3/0025: {characterised by electronic signal processing, e.g. eye models} - A61B 3/0058: {for multiple images} - A61B 3/10: Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions - A61B 3/1005: {for measuring distances inside the eye, e.g. thickness of the cornea (A61B3/11 takes precedence)} - A61B 3/102: {for optical coherence tomography [OCT]} - A61B 3/1025: {for confocal scanning} - A61B 3/13: Ophthalmic microscopes - A61B 3/14: Arrangements specially adapted for eye photography - A61B 5/0066: {Optical coherence imaging} - A61B 5/065: {Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe} - A61B 5/066: {Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging} - A61B 5/684: {Indicating the position of the sensor on the body} - A61B 5/6844: {Monitoring or controlling distance between sensor and tissue} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - G02B 21/0012: {Surgical microscopes (counterbalanced structures for surgical microscopes G02B7/001)} - G02B 21/22: Stereoscopic arrangements - G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2090/306: {using optical fibres} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/3735: {Optical coherence tomography [OCT]} - A61B 2505/05: Surgical care - A61F 9/007: Methods or devices for eye surgery - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C) ## Abstract An ophthalmic surgical system includes an imaging unit configured to generate a fundus image of an eye and a depth imaging system configured to generate a depth-resolved image of the eye. The system further includes a tracking system communicatively coupled to the imaging unit and depth imaging system, the tracking system comprising a processor and memory configured to analyze the fundus image generated by the imaging unit to determine a location of a distal tip of a surgical instrument in the fundus image, analyze the depth-resolved image generated by the depth imaging system to determine a distance between the distal tip of the surgical instrument and a retina of the eye, generate a visual indicator to overlay a portion of the fundus image, the visual indicator indicating the determined distance between the distal tip and the retina, modify the visual indicator to track a change in the location of the distal tip within the fundus image in real-time, and modify the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time. ## Claims ### Claim 1 An ophthalmic surgical system, comprising: an imaging unit configured to generate a fundus image of an eye; a depth imaging system configured to generate a depth-resolved image of the eye; a tracking system communicatively coupled to the imaging unit and depth imaging system, the tracking system comprising a processor and memory configured to: analyze the fundus image generated by the imaging unit to determine a location of a distal tip of a surgical instrument in the fundus image; analyze the depth-resolved image generated by the depth imaging system to determine a distance between the distal tip of the surgical instrument and a retina of the eye; generate a visual indicator to overlay a portion of the fundus image, the visual indicator indicating the determined distance between the distal tip and the retina; modify the visual indicator to track a change in the location of the distal tip within the fundus image in real-time; and modify the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time. ### Claim 2 (depends on 1) The ophthalmic surgical system of claim 1, wherein: the processor and memory of the tracking system are further configured to determine a distance between the distal tip of the surgical instrument and a retina of the eye based on an analysis of image pixels in the depth-resolved image. ### Claim 3 (depends on 1) The ophthalmic surgical system of claim 1, wherein: the depth-imaging system is configured to generate a depth-resolved image of the eye based on signals received by an imaging probe integrated with the surgical instrument ### Claim 4 (depends on 1) The ophthalmic surgical system of claim 1, wherein the processor and memory of the tracking system are configured to generate the visual indicator to overlay the distal tip of the surgical instrument within the fundus image. ### Claim 5 (depends on 4) The ophthalmic surgical system of claim 4, wherein the processor and memory of the tracking system are configured to modify the visual indicator to indicate the change in the distance between the distal tip of the surgical instrument and the retina by increasing or decreasing the size of the visual indicator in proportion to the change in distance between the distal tip of the surgical instrument and the retina. ### Claim 6 (depends on 1) The ophthalmic surgical system of claim 1, wherein the processor and memory of the tracking system are configured to modify the visual indicator to indicate the change in the distance between the distal tip of the surgical instrument and the retina by modifying a color of the visual indicator. ### Claim 7 (depends on 1) The ophthalmic surgical system of claim 1, wherein: the depth imaging system is an Optical Coherence Tomography (OCT) system configured to generate an OCT image of the eye, the OCT system comprising: an OCT light source operable to generate an OCT imaging beam; and a beam scanner operable to direct the OCT imaging beam; and the tracking system is configured to analyze the OCT image to determine the distance between the distal tip of the surgical instrument and the retina of the eye. ### Claim 8 (depends on 7) The ophthalmic surgical system of claim 7, wherein the processor and memory of the tracking system are further configured to cause the beam scanner to direct the OCT imaging beam to a particular region of the eye that includes the distal tip of the surgical instrument, based on the determined location of the distal tip of the surgical instrument within the fundus image. ### Claim 9 (depends on 7) The ophthalmic surgical system of claim 7, wherein the surgical instrument includes a first optical fiber configured to transmit the OCT imaging beam, and a second optical fiber configured to transmit light reflected by the eye. ### Claim 10 (depends on 1) The ophthalmic surgical system of claim 1, wherein the processor and memory of the tracking system are configured to determine the location of the distal tip of the surgical instrument in the fundus image by: generating an enhanced image of the fundus image; estimating a marker image within the enhanced image; extracting the marker image from the enhanced image; and determining a location of the marker from the image of the marker. ### Claim 11 (depends on 1) The ophthalmic surgical system of claim 1, wherein the visual indicator and fundus image are displayed in an eyepiece or on a heads-up screen. ### Claim 12 (depends on 1) The ophthalmic surgical system of claim 1, wherein the visual indicator is configurable by a user. ### Claim 13 (depends on 1) The ophthalmic surgical system of claim 1, wherein the imaging unit comprises at least one of a surgical microscope, a 2-dimensional camera, a line-scan camera, and a single detector as used in a confocal scanning ophthalmoscope. ### Claim 14 A method, comprising: generating a fundus image of an eye; generating a depth-resolved image of the eye; analyzing the fundus image to determine a location of a distal tip of a surgical instrument in the fundus image; analyzing the depth-resolved image to determine a distance between the distal tip of the surgical instrument and a retina of the eye; generating a visual indicator to overlay the distal tip of the surgical instrument within the fundus image, the visual indicator indicating the determined distance between the distal tip and the retina; modifying the visual indicator to track a change in the location of the distal tip within the fundus image in real-time; and modifying the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time. ### Claim 15 (depends on 14) The method of claim 14, wherein modifying the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time comprises increasing or decreasing the size of the visual indicator in proportion to the change in distance between the distal tip of the surgical instrument and the retina. ### Claim 16 (depends on 14) The method claim 14, wherein modifying the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time comprises modifying a color of the visual indicator. ### Claim 17 (depends on 14) The method claim 14, further comprising: directing an imaging beam of an imaging system to a particular region of the eye that includes the distal tip of the surgical instrument, based on the determined location of the distal tip of the surgical instrument within the fundus image. ### Claim 18 (depends on 14) The method claim 14, wherein analyzing the fundus image to determine a location of a distal tip of a surgical instrument in the fundus image comprises: generating an enhanced image of the fundus image; estimating a marker image within the enhanced image; extracting the marker image from the enhanced image; and determining a location of the marker from the image of the marker. ### Claim 19 (depends on 14) The method claim 14, further comprising: displaying the visual indicator in an eyepiece or on a heads-up screen. ### Claim 20 (depends on 14) The method claim 14, further comprising: receiving user input related to a type of visual indicator.