# US 2023/0414105 A1 · Image generation device, endoscopic system, and image generation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0414105 A1
- Document ID: US20230414105A1
- Status: Published application
- Kind: A1
- Application number: 18/367,625
- Filed: 2023-09-13
- Published: 2023-12-28
- Earliest claimed: 2021-03-16
- Assignees: Olympus Medical Systems Corp.
- Applicants: Olympus Medical Systems Corp.
- Inventors: Nobuyoshi ASAOKA
- Web page: https://patentgrep.com/patent/US20230414105A1

## CPC

- A61B 5/0095: {by applying light and detecting acoustic waves, i.e. photoacoustic measurements}
- A61B 5/0035: {adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography}
- A61B 5/7267: {involving training the classification device}
- A61B 8/12: in body cavities or body tracts, e.g. by using catheters
- A61B 8/4477: {using several separate ultrasound transducers or probes}
- A61B 8/5261: {combining images from different diagnostic modalities, e.g. ultrasound and X-ray}
- A61B 1/0051: {with controlled bending of insertion part}
- A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion
- A61B 1/0669: {at proximal end of an endoscope}
- A61B 1/07: using light-conductive means, e.g. optical fibres
- A61B 8/0891: {for diagnosis of blood vessels}
- A61B 8/4416: {related to combined acquisition of different diagnostic modalities, e.g. combination of ultrasound and X-ray acquisitions}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

An image generation device that generates an image of a subject includes a processor configured to generate an ultrasonic image based on a reflected ultrasound signal, generate a photoacoustic image based on a photoacoustic wave signal, and generate an estimated photoacoustic image based on the photoacoustic image and the ultrasonic image.

## Claims

### Claim 1

An image generation device that generates an image of a subject, comprising
a processor configured to
generate an ultrasonic image based on a reflected ultrasound signal,
generate a photoacoustic image based on a photoacoustic wave signal, and
generate an estimated photoacoustic image based on the photoacoustic image and the ultrasonic image.

### Claim 2 (depends on 1)

The image generation device according to claim 1, wherein
the photoacoustic image is a first photoacoustic image generated based on the photoacoustic wave signal that observed the subject at a first photoacoustic time, and
the estimated photoacoustic image is the photoacoustic image obtained by estimating the subject at a second photoacoustic time after a first period has elapsed from the first photoacoustic time.

### Claim 3 (depends on 2)

The image generation device according to claim 2, wherein the processor generates the estimated photoacoustic image based on a learned model that has learned about a relationship between the ultrasonic image and the photoacoustic image.

### Claim 4 (depends on 2)

The image generation device according to claim 2, wherein
the processor uses the first ultrasonic image and the second ultrasonic image as the ultrasonic images,
the first ultrasonic image is the ultrasonic image generated based on the reflected ultrasound signal that observed the subject at a first ultrasound time, and
the second ultrasonic image is the ultrasonic image generated based on the reflected ultrasonic signal obtained by observing the subject at a second ultrasonic time after a second period has elapsed from the first ultrasonic time.

### Claim 5 (depends on 4)

The image generation device according to claim 4, wherein
the processor generates the estimated photoacoustic image, based on the learned model that has learned about the relationship between the ultrasonic image and the photoacoustic image by using a first learning ultrasonic image, a second learning ultrasonic image, a first learning photoacoustic image, and a second learning photoacoustic image,
the first learning photoacoustic image and the second learning photoacoustic image are the photoacoustic images generated based on the photoacoustic wave signals obtained by observing the subject at intervals of the first period, and
the first learning ultrasonic image and the second learning ultrasonic image are the ultrasonic images generated based on the reflected ultrasonic signals obtained by observing the subject at intervals of the second period.

### Claim 6 (depends on 4)

The image generation device according to claim 4, wherein the processor generates the estimated photoacoustic image by image processing based on the first ultrasonic image, the second ultrasonic image, and the first photoacoustic image.

### Claim 7 (depends on 4)

The image generation device according to claim 4, wherein the second photoacoustic time approximately coincides with the second ultrasonic time.

### Claim 8 (depends on 4)

The image generation device according to claim 4, wherein the first period substantially coincides with the second period.

### Claim 9 (depends on 1)

An endoscopic system, comprising:
the image generation device according to claim 1; and
an endoscope.

### Claim 10

An image generation method for generating an image of a subject, the method comprising:
an ultrasound imaging step of generating an ultrasonic image based on a reflected ultrasound signal;
a photoacoustic image generation step of generating a photoacoustic image based on a photoacoustic wave signal; and
an estimated photoacoustic image generation step of generating an estimated photoacoustic image based on the photoacoustic image and the ultrasonic image.

### Claim 11 (depends on 10)

The image generation method according to claim 10, wherein
the photoacoustic image used in the estimated photoacoustic image generation step is a first photoacoustic image generated based on the photoacoustic wave signal obtained by observing the subject at a first photoacoustic time, and
the estimated photoacoustic image is the photoacoustic image obtained by estimating the subject at a second photoacoustic time after a first period has elapsed from the first photoacoustic time.

### Claim 12 (depends on 11)

The image generation method according to claim 11, wherein
in the estimated photoacoustic image generation step, the estimated photoacoustic image is generated based on a learned model that has learned the relationship between the ultrasonic image and the photoacoustic image.

### Claim 13

An image generation device comprising
a processor configured to
acquire an ultrasound image generated based on a reflected ultrasound signal,
obtain a photoacoustic image generated based on a photoacoustic wave signal, and
generate an estimated photoacoustic image based on the photoacoustic image and the ultrasound image.

### Claim 14

A learning model generation method of generating a learning model that
acquires a plurality of teacher data containing
a first photoacoustic image generated based on a photoacoustic wave signal obtained by observing a subject at a first time,
a first ultrasound image generated based on a reflected ultrasound signal obtained by observing the subject observed at the first time,
a second photoacoustic image generated based on the photoacoustic wave signal obtained by observing the subject at a second time after the first time, and
a second ultrasound image generated based on the reflected ultrasound signal obtained by observing the subject at the second time,
inputs
the first photoacoustic image generated based on the photoacoustic wave signal obtained by observing the subject at the first time,
the first ultrasound image generated based on reflected ultrasound signal obtained by observing the subject observed at the first time, and
the second ultrasound image generated based on the reflected ultrasound signal obtained by observing the subject at the second time, and
outputs an estimated photoacoustic image estimating the subject at the second time.
