# US 12,593,984 B2 · System, information storage medium, and information processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,593,984 B2
- Document ID: US12593984B2
- Status: Granted
- Kind: B2
- Application number: 18/237,732
- Filed: 2023-08-24
- Published: 2023-12-21
- Granted: 2026-04-07
- Earliest claimed: 2021-07-13
- Assignees: Olympus Corporation
- Applicants: Olympus Corporation
- Inventors: Masahiro Fujii; Takeshi Arai
- Web page: https://patentgrep.com/patent/US12593984B2

## CPC

- A61B 5/0093: {Detecting, measuring or recording by applying one single type of energy and measuring its conversion into another type of energy}
- G06V 40/67: {by interactive indications to the user}
- A61B 17/00: Surgical instruments, devices or methods (A61B18/00 takes precedence; dental tools A61C3/00; eye surgery A61F9/007; ear surgery A61F11/20)
- A61B 18/04: by heating (by applying electromagnetic radiation A61B18/18)
- A61B 18/12: by passing a current through the tissue to be heated, e.g. high-frequency current
- A61B 18/1445: {at the distal end of a shaft, e.g. forceps or scissors at the end of a rigid rod}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61N 7/02: Localised ultrasound hyperthermia
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/73: using feature-based methods
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/40: for the management of medical equipment or devices, e.g. scheduling maintenance or upgrades
- G16H 40/63: for local operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B 5/0538: invasively, e.g. using a catheter
- A61B 5/4848: {Monitoring or testing the effects of treatment, e.g. of medication}
- A61B 5/6847: {mounted on an invasive device}
- A61B 2017/0042: {with special provisions for gripping}
- A61B 2018/00577: {Ablation}
- A61B 2018/00702: {Power or energy}
- A61B 2018/00791: {Temperature}
- A61B 2018/00898: {Alarms or notifications created in response to an abnormal condition}
- G06T 2207/10068: Endoscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 2201/031: of internal organs
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61N: ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY (measurement of bioelectric currents A61B; surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body A61B18/00; anaesthetic apparatus in general A61M; incandescent lamps H01K; infrared radiators for heating H05B)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The system includes a memory that stores a trained model, and a processor. The processor acquires a pre-treatment image in which at least one energy device and at least one biological tissue are imaged, and a state before application of energy is imaged. The processor estimates an estimated heat diffusion region from the pre-treatment image and information regarding an energy supply amount by processing based on a trained model stored in the memory. The processor superimposes the estimated heat diffusion region on a captured image of the camera and displays the captured image with the superimposed estimated heat diffusion region.

## Claims

### Claim 1

A system comprising:
a memory configured to store a trained model that is trained to output a heat diffusion region from a training device tissue image or a training tissue image, the heat diffusion region being a range of reach of heat from the at least one energy device, the training device tissue image being an image in which at least one energy device that receives energy supply to output energy and at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged; and
a processor, wherein the processor is configured to:
acquire a pre-treatment image, in which the at least one energy device and the at least one biological tissue are imaged, in which a state before application of energy from the at least one energy device is imaged, and that is captured by a camera that captures an image of a surgical field;
acquire information regarding an energy supply amount to the at least one energy device;
estimate, based on the pre-treatment image, the information regarding the energy supply amount, and the trained model, an estimated heat diffusion region in the pre-treatment image, the estimated heat diffusion region being an estimated range of reach of energy from the at least one energy device after application of the energy based on the energy supply amount; and
perform a process of superimposing the estimated heat diffusion region on a captured image of the camera and displaying the captured image with the superimposed estimated heat diffusion region on a display;
wherein the processor further:
extracts, from the pre-treatment image, tissue information regarding a biological tissue that is imaged in the pre-treatment image,
estimates the estimated heat diffusion region based on the tissue information and the information regarding the energy supply amount;
extracts, from the pre-treatment image, device information regarding the at least one energy device that is imaged in the pre-treatment image,
estimates, based on the device information and the tissue information, the estimated heat diffusion region in a treatment target tissue to be treated by the at least one energy device,
estimates regions of the treatment target tissue and the at least one energy device in the pre-treatment image based on the tissue information and the device information,
estimates, based on the estimated regions of the treatment target tissue and the at least one energy device, at least one of a tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device as an estimation result, and
estimates the estimated heat diffusion region based on the estimation result, the tissue information, and the information regarding the energy supply amount.

### Claim 2 (depends on 1)

The system as defined in claim 1, wherein the processor estimates a gripping force of the at least one energy device based on the estimated tension and the pre-treatment image, and estimates the estimated heat diffusion region based on the estimated gripping force.

### Claim 3 (depends on 1)

The system as defined in claim 1, wherein the processor acquires a detection value of a gripping force detected by a gripping force detection sensor provided in a grip section of the at least one energy device, and estimates the estimated heat diffusion region based on the acquired detection value.

### Claim 4 (depends on 1)

The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region for each temperature or heat, superimposes the estimated heat diffusion region for the temperature or the heat that has been estimated on the pre-treatment image to display the estimated heat diffusion region.

### Claim 5 (depends on 2)

The system as defined in claim 2, wherein the processor estimates a region of a biological tissue and a region of an energy device from the pre-treatment image, and superimposes the estimated heat diffusion region on a region of a biological tissue around the at least one energy device that has been estimated to display the estimated heat diffusion region.

### Claim 6 (depends on 1)

The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of tension without estimating the tension applied to a biological tissue imaged in the pre-treatment image.

### Claim 7 (depends on 1)

The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of gripping amount without estimating the gripping amount of the at least one energy device imaged in the pre-treatment image.

### Claim 8 (depends on 1)

The system as defined in claim 1, wherein
the processor estimates, based on the pre-treatment image, tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device,
when a certainty factor in the estimation of the tension is lower than a first predetermined value, the processor estimates the estimated heat diffusion region corresponding to each of plural stages of the tension without using the estimated tension for the estimation of the estimated heat diffusion region, and
when a certainty factor in the estimation of the gripping amount is lower than a second predetermined value, the processor estimates the estimated heat diffusion region corresponding to each of plural stages of the gripping amount without using the estimated gripping amount for the estimation of the estimated heat diffusion region.

### Claim 9 (depends on 1)

The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of gripping force for gripping a biological tissue by the at least one energy device.

### Claim 10 (depends on 1)

The system as defined in claim 1, wherein the processor estimates the estimated heat diffusion region corresponding to each of plural stages of time for the at least one energy device to apply energy.

### Claim 11

A computer-readable non-transitory information storage medium storing a program for causing a computer to at least execute:
acquiring a pre-treatment image, in which at least one energy device and at least one biological tissue are imaged, in which a state before application of energy from the at least one energy device is imaged, and that is captured by a camera that captures an image of a surgical field, and acquiring information regarding an energy supply amount to the at least one energy device,
estimating an estimated heat diffusion region in the pre-treatment image by processing based on a trained model, the estimated heat diffusion region being an estimated range of reach of energy from the at least one energy device after application of the energy based on the energy supply amount, the trained model being trained to output a heat diffusion region from a training device tissue image or a training tissue image, the heat diffusion region being a range of reach of heat from the at least one energy device, the training device tissue image being an image in which the at least one energy device and the at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged, and superimposing the estimated heat diffusion region on a captured image of the camera and displaying the captured image with the superimposed estimated heat diffusion region on a display;
extracting, from the pre-treatment image, tissue information regarding a biological tissue that is imaged in the pre-treatment image,
extracting, from the pre-treatment image, device information regarding the at least one energy device that is imaged in the pre-treatment image,
estimating, based on the device information and the tissue information, the estimated heat diffusion region in a treatment target tissue to be treated by the at least one energy device,
estimating regions of the treatment target tissue and the at least one energy device in the pre-treatment image based on the tissue information and the device information,
estimating, based on the estimated regions of the treatment target tissue and the at least one energy device, at least one of a tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device as an estimation result, and
estimating the estimated heat diffusion region based on the estimation result, the tissue information, and the information regarding the energy supply amount.

### Claim 12

An information processing method, comprising:
acquiring a pre-treatment image, in which at least one energy device and at least one biological tissue are imaged, in which a state before application of energy from the at least one energy device is imaged, and that is captured by a camera that captures an image of a surgical field, and acquiring information regarding an energy supply amount to the at least one energy device,
estimating an estimated heat diffusion region in the pre-treatment image by processing based on a trained model, the estimated heat diffusion region being an estimated range of reach of energy from the at least one energy device after application of the energy based on the energy supply amount, the trained model being trained to output a heat diffusion region from a training device tissue image or a training tissue image, the heat diffusion region being a range of reach of heat from the at least one energy device, the training device tissue image being an image in which the at least one energy device and the at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged,
superimposing the estimated heat diffusion region on a captured image of the camera and displaying the captured image with the superimposed estimated heat diffusion region on a display,
extracting, from the pre-treatment image, tissue information regarding a biological tissue that is imaged in the pre-treatment image,
extracting, from the pre-treatment image, device information regarding the at least one energy device that is imaged in the pre-treatment image,
estimating, based on the device information and the tissue information, the estimated heat diffusion region in a treatment target tissue to be treated by the at least one energy device,
estimating regions of the treatment target tissue and the at least one energy device in the pre-treatment image based on the tissue information and the device information,
estimating, based on the estimated regions of the treatment target tissue and the at least one energy device, at least one of a tension applied to a biological tissue imaged in the pre-treatment image and a gripping amount of the at least one energy device as an estimation result, and
estimating the estimated heat diffusion region based on the estimation result, the tissue information, and the information regarding the energy supply amount.