# US 12,602,789 B2 · Endoscopic image segmentation method based on single image and deep learning network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,789 B2
- Document ID: US12602789B2
- Status: Granted
- Kind: B2
- Application number: 18/230,228
- Filed: 2023-08-04
- Published: 2024-11-28
- Granted: 2026-04-14
- Earliest claimed: 2023-05-26
- Assignees: University of Electronic Science and Technology of China
- Applicants: University of Electronic Science and Technology of China
- Inventors: Bing Zeng; Dingrui Liu; Haipeng Li; Yu Zeng; Xuanren Hou; Shikun Zhang; Shuaicheng Liu; Nini Rao
- Web page: https://patentgrep.com/patent/US12602789B2

## CPC

- G06T 7/12: Edge-based segmentation
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/13: Edge detection
- G06T 7/162: involving graph-based methods
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/82: using neural networks
- G16H 30/40: for processing medical images, e.g. editing
- G06T 2207/10068: Endoscopic image
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- Y02T 10/40: Engine management systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

An endoscopic image segmentation method based on a single image and a deep learning network achieves real-time and accurate segmentation for a single case, and provides support for making a treatment plan based on an endoscopic image in clinical medicine. The endoscopic image segmentation method first proposes a single image-based training set generation method to automatically generate a training set, and further proposes a lightweight deep learning network EUnet to perform feature fitting on the generated training set, to achieve endoscopic image segmentation and obtain a segmentation result of a lesion region. The endoscopic image segmentation method can significantly improve segmentation accuracy and has advantages of a small volume, high real-time performance, and easy operation. Especially in processing of endoscopic images of rare cases such as a gastric cancer and an esophageal cancer, the endoscopic image segmentation method has significant advantages in accuracy and speed, and has clinical application value.

## Claims

### Claim 1

An endoscopic image segmentation method based on a single image and a deep learning network, comprising following steps:
step 1: generating a training set by using a single image-based training set generation method based on a to-be-segmented original image, wherein the training set comprises a training image set, a segmentation label set, and an edge label set;
step 2: constructing a lightweight deep learning network EUnet, comprising an encoder and a decoder, wherein the decoder comprises a segmentation branch and an edge extraction branch;
step 3: setting a composite loss function, training the lightweight deep learning network EUnet based on the training set, and obtaining an endoscopic image segmentation model after the training; and
step 4: inputting the to-be-segmented original image into the endoscopic image segmentation model, and outputting a segmentation result by a segmentation branch of a decoder of the endoscopic image segmentation model.

### Claim 2 (depends on 1)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 1, wherein in the step 1, a specific process of the single image-based training set generation method comprises following steps:
step 1.1: obtaining a region of interest (ROI) of the to-be-segmented original image in a transcendental manner, performing foreground sampling within the ROI to obtain N circular sampling regions, and generating a sampling dataset;
step 1.2: generating M simple graph samples by using a simple graph generation method based on the sampling dataset, and generating a set of simple graph samples; and
step 1.3: randomly selecting J simple graph samples in the set of simple graph samples, and pasting the J simple graph samples onto the to-be-segmented original image in a non-overlapping manner based on random positions to obtain a training image; repeating the above operations K times to obtain K training images to constitute a training image set; and using a region for placing the simple graph samples as a foreground of a segmentation label, an edge of the region for placing the simple graph samples as a foreground of an edge label, and another region excluding the ROI as a background of the segmentation label and the edge label, to constitute the segmentation label set and the edge label set respectively.

### Claim 3 (depends on 2)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 2, wherein in the step 1.1, the foreground sampling is implemented by: extracting N circular regions with random radiuses within the ROI as the sampling regions, wherein the radius r of the sampling region meets following formulas:
r
min
=
⌊
max
⁢
(
H
,
W
)
/
128
⌋
,
r
max
=
⌊
max
⁢
(
H
,
W
)
/
8
⌋
wherein H and W represent a length and a width of the to-be-segmented image, and rmin and rmax represent minimum and maximum values of the radius r of the sampling region, and └·┘ represents a downward rounding operation.

### Claim 4 (depends on 2)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 2, wherein in the step 1.2, the simple graph generation method is implemented by:
randomly cutting and rotating the sampling region in the sampling dataset to obtain the M simple graph samples, wherein simple graphs comprise a circle and a regular triangle, and a radius {circumflex over (r)} of a randomly cut and rotated simple graph sample meets a following condition:
{circumflex over (r)}∈(rmin:rmax:l)
wherein l represents a step, and rmin and rmax represent minimum and maximum values of the radius r of the sampling region.

### Claim 5 (depends on 1)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 1, wherein in the step 2, the encoder comprises construction layers 1 to 5 connected in sequence; after an input x is input into the encoder, the construction layers 1 to 5 perform encoding and output y1 to y5 respectively; and the construction layer 1 comprises a convolutional layer conv1, the construction layer 2 comprises a maximum pooling layer and three residual (Res) blocks, wherein the maximum pooling layer and the three Res blocks are connected in sequence, the construction layer 3 comprises four Res blocks connected in sequence, the construction layer 4 comprises six Res blocks connected in sequence, and the construction layer 5 comprises three Res blocks connected in sequence;
the segmentation branch of the decoder comprises feature fusion blocks, namely, Up_Block 1 to Up_Block 4, a convolutional layer conv2, a convolutional layer conv3, and a boundary-enhance (BE) block; an input of an ith Up_Block is upi+1 and yi, upi is an output of the ith Up_Block, yi is an output of an ith construction layer, up5-y5, an output up1 of the Up_Block 1 sequentially passes through the convolutional layer conv2 and the convolutional layer conv3 to generate an image segmentation output ŝ, and the image segmentation output ŝ passes through the BE block to generate a segmented predicted edge ê′; and
the edge extraction branch of the decoder comprises edge-detect (ED) blocks 1 to 5 and an edge-attention (EA) block; the output yi of the ith construction layer sequentially passes through an ith ED block to generate an intermediate edge output {tilde over (e)}i, and the intermediate edge output {{tilde over (e)}1, {tilde over (e)}2, {tilde over (e)}3, {tilde over (e)}4, {tilde over (e)}5} then passes through the EA block to generate an edge output ê.

### Claim 6 (depends on 5)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 5, wherein the Res block is expressed as follows:
Res
⁡
(
x
)
=
H
⁡
(
x
)
+
x
,
and
⁢
⁢
H
⁡
(
x
)
=
conv
3
×
3
(
conv
3
×
3
(
x
)
)
wherein x represents the input, and conv3×3 represents a convolution operation with a 3×3 convolution kernel;
the Up_Block is expressed as follows:
up
i
=
H
⁡
(
[
upsampling
⁢
(
up
i
+
1
)
,
y
i
]
)
wherein upsampling represents a bilinear interpolation operation with a step of 2, and [,] represents a channel dimension stacking operation;
the BE block is expressed as follows:
e
^
′
=
conv
1
×
1
(
conv
3
×
3
(
conv
1
×
1
(
s
d
)
)
)
,
and
⁢
s
d
=
Laplace
⁢
(
s
^
)
wherein Laplace represents a Laplace operator, and Sa represents a calculation result of the Laplace operator;
the ED block is expressed as follows:
e
~
i
=
deconv
2
⁢
λ
×
2
⁢
λ
(
conv
1
×
1
(
Res
⁡
(
y
i
)
)
)
wherein deconv2λ×2λ represents a deconvolution operation with a 2λ×2λ convolution kernel and a step of λ, and the step factor λ constraints a deconvoluted output of the Res block to match a size of the input image; and
the EA block is expressed as follows:
e
m
=
conv
3
×
3
(
e
c
)
,
and
⁢
e
^
=
Sum
(
Attention
[
e
m
]
⊙
e
m
)
wherein ec [{tilde over (e)}1, {tilde over (e)}2, {tilde over (e)}3, {tilde over (e)}4, {tilde over (e)}5], conv3×3 represents the convolution operation with the 3×3 convolution kernel, ⊙ represents a point multiplication operation by pixel, Sum represents a summation operation by channel dimension, and Attention represents an attention operation and is expressed as: Attention [ec]=Softmax(conv3×3 (conv3×3 (conv3×3(ec)))).

### Claim 7 (depends on 1)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 1, wherein in the step 3, the composite loss function is expressed as:
L
=
μ
1
⁢
L
s
+
μ
2
⁢
L
e
+
μ
3
⁢
L
c
wherein L represents the composite loss function, Ls represents a segmentation loss function, Le represents an edge loss function, Lc represents a consistency loss function, and μ1, μ2, and μ3 respectively represent a weight factor of the segmentation loss function, a weight factor of the edge loss function, and a weight factor of the consistency loss function;
the segmentation loss function Ls is expressed as:
L
s
(
s
ˆ
,
s
)
=
λ
1
⁢
L
BCE
(
s
ˆ
,
s
)
+
λ
2
⁢
L
Dice
(
s
ˆ
,
s
)
wherein ŝ represents an image segmentation output, s represents the segmentation label, LBCE represents a binary cross entropy (BCE) loss function, LDice represents a dice loss function, and λ1 and λ2 respectively represent a weight factor of the BCE loss function and a weight factor of the dice loss function;
the edge loss function Le is expressed as:
Le(ê,e)=WBCE(ê,e)
wherein ê represents an edge output, e represents the edge label, and WBCE represents a weighted binary cross entropy (WBCE) loss function; and
the consistency loss function Lc is expressed as:
Lc(ê′,ê)=WBCE(ê′,ζ(ê))
wherein ê′ represents a predicted segmented edge, ê represents the edge output, and ζ represents a binary classification function.

### Claim 8 (depends on 1)

The endoscopic image segmentation method based on the single image and the deep learning network according to claim 1, wherein in the step 3, the training comprises a first stage and a second stage; in the first stage, the encoder is frozen, and the lightweight deep learning network EUnet is trained by using an Adam optimizer; and in the second stage, the encoder is unfrozen, and the lightweight deep learning network EUnet is retrained by using the Adam optimizer, to obtain the endoscopic image segmentation model.