# US 2015/0146946 A1 · Overlay and registration of preoperative data on live video using a portable device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0146946 A1
- Document ID: US20150146946A1
- Status: Published application
- Kind: A1
- Application number: 14/402,436
- Filed: 2013-06-28
- Published: 2015-05-28
- Earliest claimed: 2012-06-28
- Assignees: Koninklijke Pjilips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Haytham Elhawary; Aleksandra Popovic
- Web page: https://patentgrep.com/patent/US20150146946A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 7/0012: {Biomedical image inspection}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2207/30048: Heart; Cardiac
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A medical imaging system includes a computer device including a processor (114) and a memory (116) coupled to the processor. The memory stores recorded anatomical images (130) of a subject and a registration module (142) configured to align the recorded anatomical images with real-time images. A portable video display device (132) includes a camera (136) and a display screen (133). The portable video display device is configured to collect real-time images (134) of the subject to be registered with the recorded anatomical images of the subject such that a portion of the recorded anatomical images are concurrently displayed on the portable video display device registered with the real-time images. A communication link (140) is configured to permit communication and data transfer between the computer device and the portable video display device.

## Claims

### Claim 1

A medical imaging system, comprising:
a computer device including a processor and a memory coupled to the processor, the memory storing:
recorded anatomical images of a subject; and
a registration module configured to: register features in pre-operative images; register features in real-time images with corresponding features in the preoperative images; and align the recorded anatomical images with the real-time images;
a portable video display device including a camera and a display screen, the portable video display device being configured to collect the real-time images of the subject to be registered with the recorded anatomical images of the subject such that at least a portion of the recorded anatomical images is concurrently displayed on the portable video display device registered with the real-time images; and
a communication link configured to permit communication and data transfer between the computer device and the portable video display device.

### Claim 2 (depends on 1)

The system as recited in claim 1, wherein the portable video display device includes a smart phone or a tablet.

### Claim 3 (depends on 1)

The system as recited in claim 1, wherein the registration module determines and tracks common points between the recorded anatomical images and the real-time video images.

### Claim 4 (depends on 1)

The system as recited in claim 1, wherein the at least a portion of the recorded anatomical images includes an overlay concurrently displayed with the real-time video images.

### Claim 5 (depends on 1)

The system as recited in claim 1, further comprising an image processing module configured to compensate for motion of at least one of: an anatomical feature being imaged and movement of the portable video display device.

### Claim 6 (depends on 5)

The system as recited in claim 5, wherein the anatomical feature includes a beating heart and the at least a portion of the recorded anatomical images is adjusted to track motion of the beating heart.

### Claim 7 (depends on 5)

The system as recited in claim 5, further comprising a homography matrix H computed over time from a target anatomical feature in the anatomical feature being imaged, wherein the matrix H is employed to deform the recorded anatomical images to maintain registration with the real-time images.

### Claim 8 (depends on 1)

The system as recited in claim 1, wherein the recorded anatomical images include preoperative or intraoperative images collected using one or more imaging modalities.

### Claim 9 (depends on 1)

The system as recited in claim 1, further comprising an image processing module configured to provide overlay features from the recorded anatomical images to be overlaid onto the real-time images for display on the portable video display device.

### Claim 10 (depends on 9)

The system as recited in claim 9, wherein movement of the portable video display device changes the real-time images and the overlay features accordingly.

### Claim 11 (depends on 1)

The system as recited in claim 1, further comprising an arm to secure a position of the portable video display device during a procedure.

### Claim 12

A portable video display device for medical imaging, comprising:
a camera configured to collect real-time video;
a display screen configured to display the real-time video concurrently with overlay images registered with real-time images;
a processor; and
a memory coupled to the processor, the memory storing:
recorded anatomical images of a subject; and
a registration module configured to: register features in pre-operative images; register features in real-time images with corresponding features in the preoperative images; and align the recorded anatomical images with the real-time images;
wherein the overlay images include at least a portion of the stored anatomical images and are displayed on the portable video display device to assist a user in viewing anatomical features not readily observable.

### Claim 13 (depends on 12)

The device as recited in claim 12, wherein the portable video display device includes a smart phone or a tablet.

### Claim 14 (depends on 12)

The device as recited in claim 12, wherein the registration module determines and tracks common points between the recorded anatomical images and the real-time images.

### Claim 15 (depends on 12)

The device as recited in claim 12, wherein the at least a portion of the recorded anatomical images includes hidden coronary arteries during open heart surgery.

### Claim 16 (depends on 12)

The device as recited in claim 12, further comprising an image processing module configured to compensate for motion of at least one of: an anatomical feature being imaged and movement of the portable video display device.

### Claim 17 (depends on 16)

The device as recited in claim 16, wherein the anatomical feature includes a beating heart and the at least a portion of the recorded anatomical images are adjusted to track motion of the beating heart.

### Claim 18 (depends on 16)

The system as recited in claim 16, further comprising a homography matrix H computed over time from a target anatomical feature in the anatomical feature being imaged, wherein the matrix H is employed to deform the recorded anatomical images to maintain registration with the real-time images.

### Claim 19 (depends on 16)

The device as recited in claim 16, wherein movement of the portable video display device changes the real-time images and the overlay images accordingly.

### Claim 20 (depends on 12)

The device as recited in claim 12, wherein the recorded anatomical images include preoperative or intraoperative images collected using one or more imaging modalities.

### Claim 21

A method for medical imaging, comprising:
storing anatomical images of a subject;
providing a portable video display device including a camera and a display screen, the portable video display device being configured to collect real-time images of the subject;
registering features in pre-operative images;
registering features in real-time images with corresponding features in the preoperative images;
registering at least a portion of the anatomical images with the real-time images; and
concurrently displaying on the portable video display the at least a portion of the anatomical images registered with the real-time images.

### Claim 22 (depends on 21)

The method as recited in claim 21, wherein the portable video display device includes a smart phone or a tablet.

### Claim 23 (depends on 21)

The method as recited in claim 21, wherein registering includes tracking common points between the anatomical images and the real-time images.

### Claim 24 (depends on 21)

The method as recited in claim 21, further comprising generating an overlay from the at least a portion of the anatomical images; and overlaying features from the anatomical images onto the real-time images for display on the portable video display device.

### Claim 25 (depends on 21)

The method as recited in claim 21, further comprising compensating for motion of at least one of: an anatomical feature being imaged and movement of the portable video display device.

### Claim 26 (depends on 25)

The method as recited in claim 25, wherein the anatomical feature includes a beating heart and the method further comprises tracking motion of the beating heart to adjust the at least a portion of the anatomical images.

### Claim 27 (depends on 25)

The method as recited in claim 25, wherein compensating for motion includes employing a homography matrix H computed over time from a target anatomical feature in the anatomical feature being imaged, wherein the matrix H is employed to deform the recorded anatomical images to maintain registration with the real-time images.

### Claim 28 (depends on 25)

The method as recited in claim 25, wherein movement of the portable video display device changes the real-time images and the overlay features accordingly.

### Claim 29 (depends on 21)

The method as recited in claim 21, wherein the anatomical images include preoperative or intraoperative images collected using one or more imaging modalities.
