# US 11,900,596 B2 · Method, device, and storage medium for weakly-supervised universal lesion segmentation with regional level set loss

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,900,596 B2
- Document ID: US11900596B2
- Status: Granted
- Kind: B2
- Application number: 17/479,560
- Filed: 2021-09-20
- Published: 2022-11-03
- Granted: 2024-02-13
- Earliest claimed: 2021-04-14
- Assignees: Ping An Technology (Shenzhen) Co., Ltd.
- Applicants: Ping An Technology (Shenzhen) Co., Ltd.
- Inventors: Youbao Tang; Jinzheng Cai; Ke Yan; Le Lu
- Web page: https://patentgrep.com/patent/US11900596B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06V 10/82: using neural networks
- G06F 18/253: {of extracted features}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure provides a computer-implemented method, a device, and a storage medium. The method includes inputting an image into an attention-enhanced high-resolution network (AHRNet) to extract feature maps for generating a first feature map; generating a first probability map which is concatenated with the first feature map to form a concatenated first feature map, and updating the AHRNet using the first segmentation loss; generating a second feature map, and scaling the second feature map to form a third feature map; generating a second probability map which is concatenated with the third feature map to form a concatenated third feature map, and updating the AHRNet using the second segmentation loss; generating a fourth feature map, and scaling the fourth feature map to form a fifth feature map; updating the AHRNet using the third segmentation loss and the regional level set loss; and outputting the third probability map.

## Claims

### Claim 1

A lesion segmentation method for medical images, comprising:
inputting an image into an attention-enhanced high-resolution network (AHRNet) to extract feature maps at multiple scales;
generating a first feature map according to the extracted feature maps;
generating a first probability map according to the first feature map, concatenating the first probability map with the first feature map to form a concatenated first feature map, calculating a first segmentation loss based on the first probability map, and updating the AHRNet using the first segmentation loss;
generating a second feature map by up-sampling the concatenated first feature map using a deconvolutional layer, and scaling the second feature map to form a third feature map;
generating a second probability map according to the third feature map, concatenating the second probability map with the third feature map to form a concatenated third feature map, calculating a second segmentation loss based on the second probability map, and updating the AHRNet using the second segmentation loss;
generating a fourth feature map by up-sampling the concatenated third feature map using a deconvolutional layer, and scaling the fourth feature map to form a fifth feature map;
generating a third probability map according to the fifth feature map, and calculating a third segmentation loss and a regional level set loss based on the third probability map; and
updating the AHRNet using the third segmentation loss and the regional level set loss, and outputting the third probability map.

### Claim 2 (depends on 1)

The method according to claim 1, wherein generating the first feature map according to the extracted feature maps includes:
inputting each feature map of the extracted feature maps at each scale into a dual attention (DA) module of multiple DA modules, thereby forming an enhanced feature map at each scale of multiple scales:
up-sampling one or more enhanced feature maps corresponding to one or more scales among the multiple scales, such that enhanced feature maps, including the up-sampled enhanced feature maps and non-up-sampled enhanced feature maps, have a same resolution;
inputting the enhanced feature maps having the same resolution into a scale attention (SA) module to generate concatenated feature maps; and
generating the first feature map according to the concatenated feature maps.

### Claim 3 (depends on 1)

The method according to claim 1, wherein:
the first segmentation loss is calculated according to the first feature map and a corresponding pseudo mask; the second segmentation loss is calculated according to the second feature map and a corresponding pseudo mask; and the third segmentation loss is calculated according to the third feature map and a corresponding pseudo mask, wherein the corresponding pseudo mask is an ellipse for the image, fitted from four endpoints of a response evaluation criteria in solid tumors (RECIST) annotation of the image.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the regional level set loss is computed by:
ℓ
rls
=
1
❘
"\[LeftBracketingBar]"
I
′
❘
"\[RightBracketingBar]"
⁢
∑
i
∈
I
′
[
λ
1
·
p
⁡
(
i
)
·
❘
"\[LeftBracketingBar]"
q
⁡
(
i
)
-
c
1
❘
"\[RightBracketingBar]"
2
+
λ
2
·
(
1
-
p
⁡
(
i
)
)
·
❘
"\[LeftBracketingBar]"
q
⁡
(
i
)
-
c
2
❘
"\[RightBracketingBar]"
2
]
wherein λ1 and λ2 denote predefined non-negative hyper-parameters, q(i) is an intensity of a pixel i, c1 and c2 denote mean pixel intensities of inside and outside areas of a contour, p(i) denotes a probability value of the pixel i, I′ denotes a constrained region of an input image I, and |I′| denotes a number of pixels in I′.

### Claim 5 (depends on 1)

The method according to claim 1, wherein a segmentation loss is computed, for each of the first, second, and third segmentation losses, by:
seg=bce(p,g)+iou(p,g)
wherein p denotes a probability map, g denotes a pseudo mask, bce denotes a binary cross entropy loss, and iou denotes an IoU loss.

### Claim 6 (depends on 5)

The method according to claim 5, wherein bce and iou are defined as:
ℓ
bce
(
p
,
g
)
=
-
1
❘
"\[LeftBracketingBar]"
I
❘
"\[RightBracketingBar]"
⁢
∑
i
∈
I
[
g
⁡
(
i
)
⁢
log
⁡
(
p
⁡
(
i
)
)
+
(
1
-
g
⁡
(
i
)
)
⁢
log
⁡
(
1
-
p
⁡
(
i
)
)
]
ℓ
iou
(
p
,
g
)
=
1
-
(
∑
i
∈
I
g
⁡
(
i
)
⁢
p
⁡
(
i
)
)
/
(
∑
i
∈
I
g
⁡
(
i
)
+
p
⁡
(
i
)
-
g
⁡
(
i
)
⁢
p
⁡
(
i
)
)
wherein p(i) denotes a probability value of a pixel i, g(i) denotes a pseudo mask value of the pixel i, and |I| denotes a number of pixels in I.

### Claim 7

A segmentation device for medical images, comprising:
a memory, containing a computer program stored thereon; and
a processor, coupled with the memory and configured, when the computer program being executed, to perform a method including:
inputting an image into an attention-enhanced high-resolution network (AHRNet) to extract feature maps at multiple scales;
generating a first feature map according to the extracted feature maps;
generating a first probability map according to the first feature map, concatenating the first probability map with the first feature map to form a concatenated first feature map, calculating a first segmentation loss based on the first probability map, and updating the AHRNet using the first segmentation loss;
generating a second feature map by up-sampling the concatenated first feature map using a deconvolutional layer, and scaling the second feature map to form a third feature map;
generating a second probability map according to the third feature map, concatenating the second probability map with the third feature map to form a concatenated third feature map, calculating a second segmentation loss based on the second probability map, and updating the AHRNet using the second segmentation loss;
generating a fourth feature map by up-sampling the concatenated third feature map using a deconvolutional layer, and scaling the fourth feature map to form a fifth feature map;
generating a third probability map according to the fifth feature map, and calculating a third segmentation loss and a regional level set loss based on the third probability map; and
updating the AHRNet using the third segmentation loss and the regional level set loss, and outputting the third probability map.

### Claim 8 (depends on 7)

The device according to claim 7, wherein generating the first feature map according to the extracted feature maps includes:
inputting each feature map of the extracted feature maps at each scale into a dual attention (DA) module of multiple DA modules, thereby forming an enhanced feature map at each scale of multiple scales;
up-sampling one or more enhanced feature maps corresponding to one or more scales among the multiple scales, such that enhanced feature maps, including the up-sampled enhanced feature maps and non-up-sampled enhanced feature maps, have a same resolution;
inputting the enhanced feature maps having the same resolution into a scale attention (SA) module to generate concatenated feature maps; and
generating the first feature map according to the concatenated feature maps.

### Claim 9 (depends on 7)

The device according to claim 7, wherein:
the first segmentation loss is calculated according to the first feature map and a corresponding pseudo mask; the second segmentation loss is calculated according to the second feature map and a corresponding pseudo mask; and the third segmentation loss is calculated according to the third feature map and a corresponding pseudo mask, wherein the corresponding pseudo mask is an ellipse for the image, fitted from four endpoints of a response evaluation criteria in solid tumors (RECIST) annotation of the image.

### Claim 10 (depends on 7)

The device according to claim 7, wherein the regional level set loss is computed by:
ℓ
rls
=
1
❘
"\[LeftBracketingBar]"
I
′
❘
"\[RightBracketingBar]"
⁢
∑
i
∈
I
′
[
λ
1
·
p
⁡
(
i
)
·
❘
"\[LeftBracketingBar]"
q
⁡
(
i
)
-
c
1
❘
"\[RightBracketingBar]"
2
+
λ
2
·
(
1
-
p
⁡
(
i
)
)
·
❘
"\[LeftBracketingBar]"
q
⁡
(
i
)
-
c
2
❘
"\[RightBracketingBar]"
2
]
wherein λ1 and λ2 denote predefined non-negative hyper-parameters, q(i) is an intensity of a pixel i, c1 and c2 denote mean pixel intensities of inside and outside areas of a contour, p(i) denotes a probability value of the pixel i, I′ denotes a constrained region of an input image I, and |I′| denotes a number of pixels in I′.

### Claim 11 (depends on 7)

The device according to claim 7, wherein a segmentation loss is computed, for each of the first, second, and third segmentation losses, by:
seg=bce(p,g)+iou(p,g)
wherein p denotes a probability map, g denotes a pseudo mask, bce denotes a binary cross entropy loss, and iou denotes an IoU loss.

### Claim 12 (depends on 11)

The device according to claim 11, wherein bce and iou are defined as:
ℓ
bce
(
p
,
g
)
=
-
1
❘
"\[LeftBracketingBar]"
I
❘
"\[RightBracketingBar]"
⁢
∑
i
∈
I
[
g
⁡
(
i
)
⁢
log
⁡
(
p
⁡
(
i
)
)
+
(
1
-
g
⁡
(
i
)
)
⁢
log
⁡
(
1
-
p
⁡
(
i
)
)
]
ℓ
iou
(
p
,
g
)
=
1
-
(
∑
i
∈
I
g
⁡
(
i
)
⁢
p
⁡
(
i
)
)
/
(
∑
i
∈
I
g
⁡
(
i
)
+
p
⁡
(
i
)
-
g
⁡
(
i
)
⁢
p
⁡
(
i
)
)
wherein p(i) denotes a probability value of a pixel i, g(i) denotes a pseudo mask value of the pixel i, and |I| denotes a number of pixels in I.

### Claim 13 (depends on 7)

The device according to claim 7, further including:
a display screen for displaying the outputted third probability map.

### Claim 14

A non-transitory computer-readable storage medium, storing program instructions configured to be executable by a computer to cause the computer to implement operations comprising:
inputting an image into an attention-enhanced high-resolution network (AHRNet) to extract feature maps at multiple scales;
generating a first feature map according to the extracted feature maps;
generating a first probability map according to the first feature map, concatenating the first probability map with the first feature map to form a concatenated first feature map, calculating a first segmentation loss based on the first probability map, and updating the AHRNet using the first segmentation loss;
generating a second feature map by up-sampling the concatenated first feature map using a deconvolutional layer, and scaling the second feature map to form a third feature map;
generating a second probability map according to the third feature map, concatenating the second probability map with the third feature map to form a concatenated third feature map, calculating a second segmentation loss based on the second probability map, and updating the AHRNet using the second segmentation loss;
generating a fourth feature map by up-sampling the concatenated third feature map using a deconvolutional layer, and scaling the fourth feature map to form a fifth feature map;
generating a third probability map according to the fifth feature map, and calculating a third segmentation loss and a regional level set loss based on the third probability map; and
updating the AHRNet using the third segmentation loss and the regional level set loss, and outputting the third probability map.

### Claim 15 (depends on 14)

The non-transitory computer-readable storage medium according to claim 14, wherein generating the first feature map according to the extracted feature maps includes:
inputting each feature map of the extracted feature maps at each scale into a dual attention (DA) module of multiple DA modules, thereby forming an enhanced feature map at each scale of multiple scales;
up-sampling one or more enhanced feature maps corresponding to one or more scales among the multiple scales, such that enhanced feature maps, including the up-sampled enhanced feature maps and non-up-sampled enhanced feature maps, have a same resolution;
inputting the enhanced feature maps having the same resolution into a scale attention (SA) module to generate concatenated feature maps; and
generating the first feature map according to the concatenated feature maps.

### Claim 16 (depends on 14)

The non-transitory computer-readable storage medium according to claim 14, wherein:
the first segmentation loss is calculated according to the first feature map and a corresponding pseudo mask; the second segmentation loss is calculated according to the second feature map and a corresponding pseudo mask; and the third segmentation loss is calculated according to the third feature map and a corresponding pseudo mask, wherein the corresponding pseudo mask is an ellipse for the image, fitted from four endpoints of a response evaluation criteria in solid tumors (RECIST) annotation of the image.

### Claim 17 (depends on 14)

The non-transitory computer-readable storage medium according to claim 14, wherein the regional level set loss is computed by:
ℓ
rls
=
1
❘
"\[LeftBracketingBar]"
I
′
❘
"\[RightBracketingBar]"
⁢
∑
i
∈
I
′
[
λ
1
·
p
⁡
(
i
)
·
❘
"\[LeftBracketingBar]"
q
⁡
(
i
)
-
c
1
❘
"\[RightBracketingBar]"
2
+
λ
2
·
(
1
-
p
⁡
(
i
)
)
·
❘
"\[LeftBracketingBar]"
q
⁡
(
i
)
-
c
2
❘
"\[RightBracketingBar]"
2
]
wherein λ1 and λ2 denote predefined non-negative hyper-parameters, q(i) is an intensity of a pixel i, c1 and c2 denote mean pixel intensities of inside and outside areas of a contour, p(i) denotes a probability value of the pixel i, I′ denotes a constrained region of an input image I, and |I′| denotes a number of pixels in I′.

### Claim 18 (depends on 14)

The non-transitory computer-readable storage medium according to claim 14, wherein a segmentation loss is computed, for each of the first, second, and third segmentation losses, by:
seg=bce(p,g)+iou(p,g)
wherein p denotes a probability map, g denotes a pseudo mask, bce denotes a binary cross entropy loss, and iou denotes an IoU loss.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein bce and iou are defined as:
ℓ
bce
(
p
,
g
)
=
-
1
❘
"\[LeftBracketingBar]"
I
❘
"\[RightBracketingBar]"
⁢
∑
i
∈
I
[
g
⁡
(
i
)
⁢
log
⁡
(
p
⁡
(
i
)
)
+
(
1
-
g
⁡
(
i
)
)
⁢
log
⁡
(
1
-
p
⁡
(
i
)
)
]
ℓ
iou
(
p
,
g
)
=
1
-
(
∑
i
∈
I
g
⁡
(
i
)
⁢
p
⁡
(
i
)
)
/
(
∑
i
∈
I
g
⁡
(
i
)
+
p
⁡
(
i
)
-
g
⁡
(
i
)
⁢
p
⁡
(
i
)
)
wherein p(i) denotes a probability value of a pixel i, g(i) denotes a pseudo mask value of the pixel i, and |I| denotes a number of pixels in I.