# US 2015/0145953 A1 · Image completion system for in-image cutoff region, image processing device, and program therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0145953 A1
- Document ID: US20150145953A1
- Status: Published application
- Kind: A1
- Application number: 14/383,776
- Filed: 2013-03-15
- Published: 2015-05-28
- Earliest claimed: 2012-03-17
- Assignees: Waseda University; Kyushu University, National University Corporation
- Applicants: Waseda University; Kyushu University, National University Corporation
- Inventors: Masakatsu Fujie; Yo Kobayashi; Kazuya Kawamura; Hiroto Seno; Yuya Nishio; Makoto Hashizume; Satoshi Ieiri; Kazutaka Toyoda
- Web page: https://patentgrep.com/patent/US20150145953A1

## CPC

- A61B 1/00009: {of image signals during a use of endoscope}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/018: for receiving instruments
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G02B 23/2415: {Stereoscopic endoscopes}
- G02B 23/2484: {Arrangements in relation to a camera or imaging device (processing or control of video signals generated by an endoscope H04N23/60, H04N23/70)}
- G06T 7/33: using feature-based methods
- A61B 1/313: for introducing through surgical openings, e.g. laparoscopes
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- G06T 2207/10012: Stereo images
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10048: Infrared image
- G06T 2207/10068: Endoscopic image
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30208: Marker matrix
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image completion system includes a first endoscope for obtaining an operating field image, a second endoscope for imaging the object space in a line-of-sight direction different from that of the first endoscope to obtain a completing image, a first device for measuring separating distances between a reference point and a large number of set points that are set in the object space, a second device for measuring three-dimensional positions of the endoscopes, and an image processing device for obtaining, image information on cutoff regions in the object space that are hidden on the depth sides of surgical instruments by imaging the surgical instruments in the operating field image together with the object space, and for replacing image information on the surgical instruments S in the operating field image with the obtained image information or superimposing the obtained image information onto the image information on the surgical instruments.

## Claims

### Claim at position 1

-7. (canceled)

### Claim 8

An image completion system for an in-image cutoff region, comprising:
a main imaging device for obtaining a main image in which an object space to be monitored is imaged;
a completing-purpose imaging device for obtaining a completing image used for completing the main image by imaging the object space in a line-of-sight direction different from that of the main imaging device; and
an image processing device for completing a portion of the main image with the completing image on the basis of three-dimensional positions of set points which are set in the object space and three-dimensional positions of the main imaging device and the completing-purpose imaging device, wherein
the main imaging device and the completing-purpose imaging device are provided so as to image the object space almost simultaneously, and
the image processing device obtains, from the completing image, image information on a cutoff region in the object space that is hidden behind on a depth side of a member having a known shape by imaging the member in the main image together with the object space, on the basis of information on the three-dimensional positions that are continuously detected, and replaces image information on the member in the main image with the obtained image information or superimposes the obtained image information onto the image information on the member in the main image so as to generate a composite image in which the cutoff region is completed with the completing image.

### Claim 9 (depends on 8)

The image completion system for an in-image cutoff region according to claim 8, wherein
the image processing device includes:
set point position identifying means for identifying, with respect to the set points, sets of in-screen coordinates in a screen coordinate system in the main image and sets of in-screen coordinates in a screen coordinate system in the completing image on the basis of a detection result of the three-dimensional positions;
completing image transforming means for generating, on the basis of the sets of in-screen coordinates, a transformed image in which pieces of image information on points in the completing image are moved in a screen of the completing image such that the completing image is converted into that in a line-of-sight direction of the main imaging device;
cutoff region identifying means for identifying a position of the cutoff region in the main image; and
composite image generating means for generating the composite image by replacing image information on the cutoff region with image information on a corresponding region that corresponds to the cutoff region in the transformed image or superimposing the image information on the corresponding region onto the image information on the cutoff region.

### Claim 10 (depends on 9)

The image completion system for an in-image cutoff region according to claim 9, wherein the completing image transforming means generates the transformed image by moving the pieces of image information on the completing image such that the set points in the completing image match sets of in-screen coordinates of the same set points existing in the main image.

### Claim 11

An image processing device for performing a process to compose a main image of an object space to be monitored that is obtained by a main imaging device and a completing image of the object space that is imaged by a completing-purpose imaging device at the same time in a line-of-sight direction different from the main image so as to complete, when a member having a known shape is imaged in the main image together with the object space, image information on a cutoff region in the main image that is cut off by at least a portion of the member with image information on the completing image, the image processing device comprising:
set point position identifying means for identifying, with respect to set points which are set in the object space, sets of in-screen coordinates in a screen coordinate system of the main image and sets of in-screen coordinates in a screen coordinate system of the completing image, from three-dimensional positions of the set points and three-dimensional positions of the main imaging device and the completing-purpose imaging device;
completing image transforming means for generating, on the basis of the sets of in-screen coordinates, a transformed image in which pieces of image information on points in the completing image are moved in a screen of the completing image such that the completing image is converted into that in a line-of-sight direction of the main image;
cutoff region identifying means for identifying a position of the cutoff region in the main image; and
composite image generating means for generating a composite image in which a cutoff region in the main image is completed with the completing image by replacing image information on the cutoff region with image information on a corresponding region that corresponds to the cutoff region in the transformed image or superimposing the image information on the corresponding region onto the image information on the cutoff region.

### Claim 12 (depends on 8)

The image completion system for an in-image cutoff region according to claim 8, wherein at least three of the set points are set in the object space.

### Claim 13 (depends on 8)

The image completion system for an in-image cutoff region according to claim 8, wherein the main imaging device and the completing-purpose imaging device are provided so as to independently move to image.

### Claim 14 (depends on 8)

The image completion system for an in-image cutoff region according to claim 8, further comprising:
a distance measuring device for measuring separating distances between the set points and a predetermined reference point; and
a three-dimensional position measuring device for measuring three-dimensional positions of the main imaging device and the completing-purpose imaging device, wherein
in the image processing device, a portion of the main image is completed with the completing image on the basis of measurement results from the distance measuring device and the three-dimensional position measuring device.
