# US 2021/0330393 A1 · Using real-time images for augmented-reality visualization of an ophthalmology surgical tool

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0330393 A1
- Document ID: US20210330393A1
- Status: Published application
- Kind: A1
- Application number: 16/856,696
- Filed: 2020-04-23
- Published: 2021-10-28
- Earliest claimed: 2020-04-23
- Assignees: Johnson & Johnson Surgical Vision, Inc.
- Applicants: Johnson & Johnson Surgical Vision, Inc.
- Inventors: Assaf Govari; Vadim Gliner
- Web page: https://patentgrep.com/patent/US20210330393A1

## CPC

- A61B 3/10: Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B 1/000094: {extracting biological structures}
- A61B 1/01: Guiding arrangements therefore
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- G02B 27/017: {Head mounted}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/38: Registration of image sequences
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A61B 5/06: Devices, other than using radiation, for detecting or locating foreign bodies {; Determining position of diagnostic devices within or on the body of the patient}
- A61B 5/6844: {Monitoring or controlling distance between sensor and tissue}
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2059: {Mechanical position encoders}
- A61B 2034/2074: {Interface software}
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/30041: Eye; Retina; Ophthalmic
- G06T 2210/41: Medical
- G06T 2211/428: Real-time
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system includes a processor and a display. The processor is configured to: (a) receive, from a camera inserted into an eye of a patient, at least an optical image of at least a region-of-interest (ROI) of the eye, (b) receive, from a position tracking system (PTS), a position signal indicative of a position of a medical instrument treating the eye, (c) register the optical image and the PTS in a common coordinate system, and (d) estimate the position of the medical instrument in the optical image. The display is configured to visualize at least the ROI and the medical instrument.

## Claims

### Claim 1

A system, comprising:
a processor, which is configured to:
receive, from a camera inserted into an eye of a patient, at least an optical image of at least a region-of-interest (ROI) of the eye;
receive, from a position tracking system (PTS), a position signal indicative of a position of a medical instrument treating the eye;
register the optical image and the PTS in a common coordinate system; and
estimate the position of the medical instrument in the optical image; and
a display, which is configured to visualize at least the ROI and the medical instrument.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the optical image comprises a real-time (RT) image.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the camera is coupled with the medical instrument and further comprising a second camera coupled to a tool for inserting the second camera into the eye, and wherein the at least optical image comprises at least one of (a) a first optical image received from the first camera and (b) a second optical image received from the second camera.

### Claim 4 (depends on 1)

The system according to claim 1, wherein the processor is configured to receive at least an anatomical image of at least the ROI, and wherein at least part of the medical instrument is obstructed by a blocking element, in at least one of the anatomical image and the optical image, and wherein the processor is configured to display the position of the medical instrument unobstructed.

### Claim 5 (depends on 4)

The system according to claim 4, wherein, when at least part of the medical instrument is obstructed, the processor is configured to visualize the medical instrument overlaid on at least one of the anatomical image and the optical image.

### Claim 6 (depends on 4)

The system according to claim 4, wherein the anatomical image comprises at least one of: (i) an optical-based anatomical image, and (ii) a computerized tomography-based anatomical image.

### Claim 7 (depends on 4)

The system according to claim 4, wherein the display comprises an augmented reality display configured to display an augmented reality image, and wherein the processor is configured to simultaneously display, on the augmented reality display, the optical image on a first section of the augmented reality image, and the anatomical image on a second section of the augmented reality image.

### Claim 8 (depends on 4)

The system according to claim 4, wherein, (i) when at least part of the medical instrument is obstructed in the anatomical image, the processor is configured to display the ROI and the medical instrument of the optical image, and (ii) when at least part of the medical instrument is obstructed in the optical image, the processor is configured to display the ROI and the medical instrument of the anatomical image.

### Claim 9 (depends on 4)

The system according to claim 4, wherein when at least part of the medical instrument is obstructed in the anatomical image and in the optical image, the processor is configured, based on the position signal, to visualize the estimated position of the obstructed part of the anatomical image, in at least one of the anatomical image and the optical image.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the common coordinate system comprises a coordinate system of the PTS.

### Claim 11

A method, comprising:
receiving, from a camera inserted into an eye of a patient, at least an optical image of at least a region-of-interest (ROI) of the eye;
receiving, from a position tracking system (PTS), a position signal indicative of a position of a medical instrument treating the eye;
registering the optical image and the PTS in a common coordinate system;
estimating the position of the medical instrument in the optical image; and
visualizing at least the ROI and the medical instrument.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the optical image comprises a real-time (RT) image.

### Claim 13 (depends on 11)

The method according to claim 11, wherein receiving at least the optical image comprises receiving at least: (a) a first optical image from a first camera coupled with the medical instrument, and (b) a second optical image from a second camera coupled with a tool for inserting the second camera into the eye.

### Claim 14 (depends on 11)

The method according to claim 11, further comprising receiving at least an anatomical image of at least the ROI, and wherein at least part of the medical instrument is obstructed by a blocking element, in at least one of the anatomical image and the optical image, and wherein visualizing at least the ROI and the medical instrument comprises displaying the position of the medical instrument unobstructed.

### Claim 15 (depends on 14)

The method according to claim 14, wherein, when at least part of the medical instrument is obstructed, and wherein visualizing at least the ROI and the medical instrument comprises visualizing the medical instrument overlaid on at least one of the anatomical image and the optical image.

### Claim 16 (depends on 14)

The method according to claim 14, wherein receiving at least the anatomical image comprises receiving at least one of: (i) an optical-based anatomical image, and (ii) a computerized tomography-based anatomical image.

### Claim 17 (depends on 14)

The method according to claim 14, wherein visualizing at least the ROI and the medical instrument comprises simultaneously displaying, on an augmented reality display, the optical image on a first section of an augmented reality image, and the anatomical image on a second section of the augmented reality image.

### Claim 18 (depends on 14)

The method according to claim 14, wherein, (i) when at least part of the medical instrument is obstructed in the anatomical image, visualizing at least the ROI and the medical instrument comprises displaying the ROI and the medical instrument of the optical image, and (ii) when at least part of the medical instrument is obstructed in the optical image, visualizing at least the ROI and the medical instrument comprises displaying the ROI and the medical instrument of the anatomical image.

### Claim 19 (depends on 14)

The method according to claim 14, wherein when at least part of the medical instrument is obstructed in the anatomical image and in the optical image, visualizing at least the ROI and the medical instrument comprises, based on the position signal, visualizing the estimated position of the obstructed part of the anatomical image, in at least one of the anatomical image and the optical image.

### Claim 20 (depends on 11)

The method according to claim 11, wherein the common coordinate system comprises a coordinate system of the PTS.
