# US 2015/0173846 A1 · Microsurgery system for displaying in real time magnified digital image sequences of an operated area

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0173846 A1
- Document ID: US20150173846A1
- Status: Published application
- Kind: A1
- Application number: 14/642,201
- Filed: 2015-03-09
- Published: 2015-06-25
- Earliest claimed: 2012-09-10
- Assignees: Elbit Systems Ltd.
- Applicants: Elbit Systems Ltd.
- Inventors: Ron SCHNEIDER; Abraham ZEITOUNY
- Web page: https://patentgrep.com/patent/US20150173846A1

## CPC

- A61B 1/000094: {extracting biological structures}
- A61B 1/000095: {for image enhancement}
- A61B 1/00042: {for mechanical operation}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 5/0066: {Optical coherence imaging}
- A61B 5/748: {Selection of a region of interest, e.g. using a graphics tablet}
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- A61B 5/1114: {Tracking parts of the body}
- A61B 90/20: Surgical microscopes characterised by non-optical aspects
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/368: {changing the image on a display according to the operator's position}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G02B 2027/0147: {comprising a device modifying the resolution of the displayed image}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G05B 2219/35503: Eye tracking associated with head mounted display to detect eye position
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A system captures and displays video of surgeries. The system may include at least one digital image sensor optically coupled to one or more lenses and configured to capture a video sequence of a scene in a surgery; at least one interface configured to receive at least one region on interest (ROI) of the captured video sequence; an electronic display, selected so that at least one of the digital image sensors has a pixel resolution which is substantially greater than the pixel resolution of the electronic display; and a computer processor configured to: receive the at least one captured video sequence and the at least one received ROI and display over the at least one electronic display a portion of the captured video sequence based on the at least one selected ROI.

## Claims

### Claim 1

A microsurgery system comprising:
A Head Mounted Display (HMD) configured to display to a user in real-time magnified image sequences of a Region of Interest (ROI) of an operated area;
at least one camera configured to acquire operated-area image sequences of said operated area, a size of said operated area being larger than a size of said ROI, said at least one camera being suspended above said operated area and being mechanically and optically disconnected from said HMD;
a memory unit configured to store said operated-area image sequences; and
a processing device coupled with said HMD, said at least one camera and with said memory unit, said processing device being configured to receive an input indicating said ROI, and configured to produce said magnified image sequences of said ROI from said operated-area image sequences.

### Claim 2 (depends on 1)

The microsurgery system according to claim 1, wherein said processing device is configured to grab a full image out of a sensor of said at least one camera, and to digitally crop and resize said ROI.

### Claim 3 (depends on 1)

The microsurgery system according to claim 1, wherein said processing device is configured to grab said ROI out of a sensor of said at least one camera by at least one of the list consisting of:
addressing only pixels corresponding to said ROI; and
under-sampling said sensor.

### Claim 4 (depends on 1)

The microsurgery system of claim 1, further comprising a tracker configured to track a line of sight (LOS) of said user, wherein said processing device is configured to determine said ROI according to said LOS.

### Claim 5 (depends on 4)

The microsurgery system according to claim 4, wherein said tracker is an eye tracker.

### Claim 6 (depends on 4)

The microsurgery system according to claim 4, wherein said image sequences of said ROI are modified at least partially based on spatial movements of said HMD as detected by said tracker.

### Claim 7 (depends on 1)

The microsurgery system according to claim 1, wherein said at least one camera is configured to carry out automatic focus based on said ROI.

### Claim 8 (depends on 1)

The microsurgery system according to claim 1, further comprising an array of controllable light sources facing said operated-area, and wherein said controllable light sources are dynamically adjusted based on a dynamic analysis of an image histogram of said image sequences to achieve at least one effect selected from:
reduce specular reflections; and
increase the dynamic range of the captured scene by of said at least one camera.

### Claim 9 (depends on 1)

The microsurgery system according to claim 1, wherein said HMD displays said magnified image sequences of said ROI to said user at an angular resolution which is equal to or smaller than the angular resolution of a human eye.

### Claim 10 (depends on 1)

The microsurgery system according to claim 1, wherein said HMD includes a first region and a second region, an angular resolution of said first region being numerically smaller than 0.000582 radians and an angular resolution of said second region being numerically larger than that of said first region, and wherein said first region has a field of view of at least 2° and is substantially centered around said LOS of said user.

### Claim 11 (depends on 1)

The microsurgery system of claim 1, wherein said processing device crops a ROI portion of said operated-area image sequences respective of said ROI, and resizes said ROI portion for producing said magnified image sequences of said ROI.

### Claim 12 (depends on 1)

The microsurgery system of claim 1, wherein said at least one camera includes two image sensors, and further includes a folding optical system located in front of said two image sensors, said folding optical system including at least a pair of fixed mirrors and a pair of moveable mirrors, wherein at least one of said moveable mirrors is configured to move away and toward another one of said fixed mirrors.

### Claim 13 (depends on 15)

The microsurgery system of claim 15, wherein each of said movable mirrors is configured to move toward and away from another one of said moveable mirrors, and wherein each of said movable mirrors is configured to rotate about an axis perpendicular to an optical axis of said microsurgery system and perpendicular to an axis extending between said movable mirrors, thereby said movable mirrors enable said microsurgery system to adjust a parallax of an object in an image acquired by a first one of said image sensors and an image acquired by a second one of said image sensors.

### Claim 14 (depends on 1)

The microsurgery system of claim 1, wherein said at least one camera includes an image sensor, a secondary sensor and a beam splitter directing light from said operated area toward said image sensor and toward said secondary image sensor.

### Claim 15 (depends on 1)

The microsurgery system of claim 1, further including a robotic arm, wherein said at least one camera is mounted on said robotic arm.

### Claim 16 (depends on 15)

The microsurgery system of claim 15, wherein said robotic arm is enslaved to a LOS of said HMD.

### Claim 17 (depends on 15)

The microsurgery system of claim 15, wherein said robotic arm is enslaved to a medical tool operated by said user.

### Claim 18 (depends on 1)

The microsurgery system of claim 1, further comprising a focus-control 2D sensor array configured to receive an image captured by a selected one of said at least one camera, said focus-control sensor being tilted with respect to an optical axis of said at least one camera, wherein said processing device is configured to determine a location in said focus-control sensor in which an image is focused, and accordingly to adjust focus characteristics of said selected one of said at least one camera.

### Claim 19 (depends on 1)

The microsurgery system of claim 1, further comprising a subcutaneous imaging module including an infrared (IR) light source, a first optical transform system, a second optical transform system, a spatial frequency filter and an image sensor, said spatial frequency filter being located between said first optical transform system and said second optical transform system and said image sensor is located after said second optical transform system, wherein said IR light source illuminates said operated area, said first optical transform system is configured to transform an IR image of said operated area into a frequency image of said operated area, said spatial frequency filter being configured to direct selected frequencies of said frequency image toward said second optical transform system, said second optical transform system being configured to transform said frequency image back into said operated area image.

### Claim 20 (depends on 1)

The microsurgery system according to claim 1, wherein said HMD is a see-through display.

### Claim 21 (depends on 20)

The microsurgery system of claim 20, wherein said HMD includes a shutter configured for controlling a transparency of at least a portion of said HMD see through display.

### Claim 22 (depends on 21)

The microsurgery system of claim 21, wherein said shutter is configured to automatically change said transparency according to orientation and/or position changes of said HMD.

### Claim 23 (depends on 1)

The microsurgery system of claim 1, wherein said processing device is configured to determine a virtual Depth of Field (DOF) out of a DOF of said at least one camera, said processing device being configured to determine which elements of said operated area image sequences are out of said virtual DOF, and wherein said processing device is configured to blur said elements of said operated area image sequences that are out of said virtual DOF.

### Claim 24 (depends on 1)

The microsurgery system of claim 1, further comprising an OCT module.

### Claim 25 (depends on 1)

The microsurgery system of claim 1, further comprising a light source for illuminating said operated area, wherein said light source is configured to transmit light pulses according to a selected time scheme, and wherein said at least one camera is synchronized with said time scheme of said light source.

### Claim 26 (depends on 1)

The microsurgery system of claim 1, further comprising an additional HMD, wherein said processing device is further configured to produce additional magnified image sequences of an additional ROI from said operated-area image sequences, and wherein said additional HMD is configured to display to an additional user in real time said additional magnified image sequences of said additional ROI.

### Claim 27 (depends on 1)

The microsurgery system of claim 1, wherein said processing device is further configured to produce a Three Dimensional (3D) model of said ROI from said operated-area image sequences.

### Claim 28 (depends on 1)

The microsurgery system of claim 1, further comprising at least two light sources, each of said at least two light sources being configured to produce illumination at a respective spectral band.

### Claim 29 (depends on 1)

The microsurgery system of claim 1, wherein said at least one camera is configured to operate without an optical zoom mechanism.

### Claim 30 (depends on 1)

The microsurgery system of claim 1, further comprising a lighting system including a light source and a shutter module, said shutter module being positioned between said light source and said operated area, said light source being configured to produce an illumination beam for illuminating said operated area, said shutter module being configured for selectively attenuating spatial portions of said illumination beam.

### Claim 31

A method for displaying in real time magnified digital image sequences of an operated area, the method comprising the following procedures:
acquiring operated-area image sequences of said operated area from a viewpoint suspended above said operated area;
receiving an input indicating a Region of Interest (ROI) of said operated area, a size of said operated area being larger than a size of said ROI;
producing magnified image sequences of said ROI from said operated-area image sequences; and
displaying in real-time said magnified image sequences of said ROI to a Line of Sight (LOS) of a user,
wherein said LOS of said user is independent of said viewpoint.
