# US 2024/0005495 A1 · Image processing device, method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005495 A1
- Document ID: US20240005495A1
- Status: Published application
- Kind: A1
- Application number: 18/336,934
- Filed: 2023-06-16
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: FUJIFILM Corporation
- Applicants: FUJIFILM Corporation
- Inventors: Sadato AKAHORI
- Web page: https://patentgrep.com/patent/US20240005495A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 19/003: {Navigation within 3D models or images}
- G06T 2207/10068: Endoscopic image
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30061: Lung
- G06T 2210/41: Medical
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A processor acquires a three-dimensional image of a subject, acquires a real endoscopic image in a lumen structure of the subject, which is picked up by an endoscope inserted into the lumen structure of the subject, derives a virtual viewpoint in the three-dimensional image of the endoscope using the real endoscopic image and the three-dimensional image, acquires a radiation image of the subject in which the endoscope is inserted into the lumen structure, performs registration between the radiation image and the three-dimensional image, and superimposes and displays at least a part of the lumen structure included in the three-dimensional image, on the radiation image, based on the virtual viewpoint and a result of the registration.

## Claims

### Claim 1

An image processing device comprising:
at least one processor,
wherein the processor is configured to:
acquire a three-dimensional image of a subject;
acquire a real endoscopic image in a lumen structure of the subject, which is picked up by an endoscope inserted into the lumen structure of the subject;
derive a virtual viewpoint in the three-dimensional image of the endoscope using the real endoscopic image and the three-dimensional image;
acquire a radiation image of the subject in which the endoscope is inserted into the lumen structure;
perform registration between the radiation image and the three-dimensional image; and
superimpose and display at least a part of the lumen structure included in the three-dimensional image, on the radiation image, based on the virtual viewpoint and a result of the registration.

### Claim 2 (depends on 1)

The image processing device according to claim 1,
wherein the processor is configured to:
detect a position of the endoscope from the radiation image; and
perform the registration by setting a projection condition of the three-dimensional image such that a projection position in a case in which the virtual viewpoint is projected onto the radiation image matches the position of the endoscope.

### Claim 3 (depends on 2)

The image processing device according to claim 2,
wherein the processor is configured to further set the projection condition such that a specific anatomical structure included in the radiation image matches the specific anatomical structure included in a projection image derived by projecting the three-dimensional image onto the radiation image.

### Claim 4 (depends on 2)

The image processing device according to claim 2,
wherein the registration includes non-rigid registration.

### Claim 5 (depends on 1)

The image processing device according to claim 1,
wherein the processor is configured to:
specify at least a part of the lumen structure included in the three-dimensional image to be projected onto the radiation image based on a position and an orientation of the virtual viewpoint; and
superimpose and display the specified part of the lumen structure on the radiation image.

### Claim 6

An image processing method comprising:
acquiring a three-dimensional image of a subject;
acquiring a real endoscopic image in a lumen structure of the subject, which is picked up by an endoscope inserted into the lumen structure of the subject;
deriving a virtual viewpoint in the three-dimensional image of the endoscope using the real endoscopic image and the three-dimensional image;
acquiring a radiation image of the subject in which the endoscope is inserted into the lumen structure;
performing registration between the radiation image and the three-dimensional image; and
superimposing and displaying at least a part of the lumen structure included in the three-dimensional image, on the radiation image, based on the virtual viewpoint and a result of the registration.

### Claim 7

A non-transitory computer-readable storage medium that stores an image processing program causing a computer to execute a process comprising:
acquiring a three-dimensional image of a subject;
acquiring a real endoscopic image in a lumen structure of the subject, which is picked up by an endoscope inserted into the lumen structure of the subject;
deriving a virtual viewpoint in the three-dimensional image of the endoscope using the real endoscopic image and the three-dimensional image;
acquiring a radiation image of the subject in which the endoscope is inserted into the lumen structure;
performing registration between the radiation image and the three-dimensional image; and
superimposing and displaying at least a part of the lumen structure included in the three-dimensional image, on the radiation image, based on the virtual viewpoint and a result of the registration.
