# US 2022/0036124 A1 · Image processing method and device, and computer-readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036124 A1
- Document ID: US20220036124A1
- Status: Published application
- Kind: A1
- Application number: 17/107,005
- Filed: 2020-11-30
- Published: 2022-02-03
- Earliest claimed: 2020-07-31
- Assignees: Sensetime Group Limited
- Applicants: Sensetime Group Limited
- Inventors: Yunhe Gao; Rui Huang; Zhiqiang Hu; Hongsheng Li
- Web page: https://patentgrep.com/patent/US20220036124A1

## CPC

- G06T 7/11: Region-based segmentation
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 7/136: involving thresholding
- G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56)
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06V 10/82: using neural networks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The disclosure provides an image processing method and device, a computer-readable storage medium. The method includes: performing feature extraction on an image to be processed through a primary image segmentation model to obtain a feature map of the image to be processed; and performing image segmentation on a segmentation target according to the feature map through a target image segmentation model to obtain a target segmentation result, herein the target image segmentation model is obtained from an adversarial training with an image encoding model, and the image encoding model is used for placing a shape constraint on the target segmentation result. The disclosure may improve an accuracy of the image segmentation.

## Claims

### Claim 1

An image processing method, comprising:
performing feature extraction on an image to be processed through a primary image segmentation model to obtain a feature map of the image to be processed; and
performing image segmentation on a segmentation target according to the feature map through a target image segmentation model to obtain a target segmentation result, wherein the target image segmentation model is obtained from an adversarial training with an image encoding model, and the image encoding model is used for placing a shape constraint on the target segmentation result.

### Claim 2 (depends on 1)

The method of claim 1, further comprising: before performing the image segmentation on the segmentation target according to the feature map through the target image segmentation model to obtain the target segmentation result,
obtaining a training sample image set, wherein the training sample image set comprises at least one training sample image corresponding to the segmentation target, and the training sample image has a labeled segmentation target;
performing the image segmentation on each training sample image through an initial target image segmentation model to obtain a predicted segmented image; and
performing the adversarial training of the initial target image segmentation model in combination with an initial image encoding model according to the labeled segmentation target and the predicted segmented image to obtain the target image segmentation model.

### Claim 3 (depends on 2)

The method of claim 2, wherein performing the adversarial training of the initial target image segmentation model in combination with the initial image encoding model according to the labeled segmentation target and the predicted segmented image to obtain the target image segmentation model comprises:
encoding the labeled segmentation target and the predicted segmented image using the initial image encoding model to obtain a first space shape code and a second space shape code, wherein the first space shape code is a low-dimensional manifold space representation of the labeled segmentation target, and the second space shape code is a low-dimensional manifold space representation of the predicted segmented image;
calculating a code distance between the first space shape code and the second space shape code through a preset minimum square error loss function to obtain a prediction error; and
updating network parameters of the initial target image segmentation model based on a shape constraint of the prediction error, iterating the adversarial training process and terminating the adversarial training process until a final prediction error is less than a preset segmentation error threshold to obtain the target image segmentation model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the initial image encoding model comprises a decoding sub-model, and after calculating the code distance between the first space shape code and the second space shape code through the preset minimum square error loss function to obtain the prediction error, the method further comprises:
decoding and performing image reconstruction on the first space shape code and the second space shape code respectively through the decoding sub-model to obtain a first reconstructed image and a second reconstructed image; and
combining, through a preset reconstruction loss function, an error between the labeled segmentation target and the first reconstructed image with an error between the predicted segmented image and the second reconstructed image to obtain a reconstruction error of the initial image encoding model.

### Claim 5 (depends on 4)

The method of claim 4, wherein updating the network parameters of the initial target image segmentation model based on the shape constraint of the prediction error, iterating the adversarial training process and terminating the adversarial training process until the final prediction error is less than the preset segmentation error threshold to obtain the target image segmentation model comprises:
updating the network parameters of the initial target image segmentation model using the prediction error in an M-th training, wherein M is an integer greater than or equal to 1;
obtaining a comprehensive encoding error based on the reconstruction error and the prediction error in an (M+1)-th training;
updating the network parameters of the initial image encoding model using the comprehensive encoding error; and
iterating the adversarial training process until the final prediction error is less than the preset encoding error threshold to obtain a target encoding model and the target image segmentation model.

### Claim 6 (depends on 3)

The method of claim 3, wherein the initial image encoding model comprises an encoding sub-model comprising at least one convolution layer and a fully connected layer,
wherein encoding the labeled segmentation target and the predicted segmented image using the initial image encoding model to obtain the first space shape code and the second space shape code comprises:
squeezing the training sample image and the predicted segmented image through the at least one convolution layer according to a preset step size of the at least one convolution layer, and encoding the labeled segmentation target in a squeezed training sample image and a squeezed predicted segmented image through the fully connected layer to obtain the first space shape code and the second space shape code.

### Claim 7 (depends on 4)

The method of claim 4, wherein the decoding sub-model comprises at least one de-convolution layer, a Rectified Linear Unit (ReLU) and a batch normalization layer.

### Claim 8 (depends on 1)

The method of claim 1, wherein performing the image segmentation on the segmentation target according to the feature map through the target image segmentation model to obtain the target segmentation result comprises:
locating the segmentation target through a target locating model according to the feature map to obtain a localization result of the segmentation target; and
performing the image segmentation on the segmentation target through the target image segmentation model according to the localization result to obtain the target segmentation result.

### Claim 9 (depends on 1)

The method of claim 1, wherein the primary image segmentation model comprises a squeeze-and-excitation (SE) module, wherein performing the feature extraction on the image to be processed through the primary image segmentation model to obtain the feature map of the image to be processed comprises:
performing down-sampling on the image to be processed twice to obtain a down-sampled image;
performing convolutional processing on the down-sampled image according to a preset convolution layer dilation rate to obtain a convolutional result;
performing residue and SE processing on the convolutional result through the SE module to obtain an excitation result; and
performing multi-scale feature extraction and de-convolutional processing on the excitation result to obtain the feature map of the image to be processed.

### Claim 10 (depends on 8)

The method of claim 8, wherein the feature map comprises N layers of feature maps, N being an integer greater than 1, wherein locating the segmentation target through the target locating model according to the feature map to obtain the localization result comprises:
performing feature recognition and normalization processing on an N-th layer of feature map through the target locating model to obtain a probability that each pixel in the N-th layer of feature map is in a center location of the segmentation target; and
performing the image segmentation on the N-th layer of feature map according to the probability to obtain the localization result of the segmentation target.

### Claim 11 (depends on 10)

The method of claim 10, wherein performing the image segmentation on the segmentation target through the target image segmentation model according to the localization result to obtain the target segmentation result comprises:
predicting a shape of the segmentation target through the target image segmentation model to obtain a cropping range of the segmentation target; and
performing cropping processing on the N-th layer of feature map according to the localization result of the segmentation target and the cropping range to obtain the target segmentation result.

### Claim 12 (depends on 8)

The method of claim 8, further comprising: after performing the feature extraction on the image to be processed through the primary image segmentation model to obtain the feature map of the image to be processed,
performing location and segmentation processing on the feature map through the primary image segmentation model according to a preset primary segmentation target to obtain a primary segmentation result corresponding to the primary segmentation target,
wherein after performing the image segmentation on the segmentation target through the target image segmentation model according to the localization result to obtain the target segmentation result, the method further comprises:
fusing the primary segmentation result and the target segmentation result in the image to be processed to obtain an image segmentation result of the image to be processed.

### Claim 13

An image processing device, comprising:
a memory storing processor-executable instructions; and
a processor arranged to execute the stored processor-executable instructions to perform operations of:
performing feature extraction on an image to be processed to obtain a feature map of the image to be processed; and
performing image segmentation on a segmentation target according to the feature map to obtain a target segmentation result, wherein a target image segmentation model is obtained from an adversarial training with an image encoding model, and the image encoding model is used for placing a shape constraint on the target segmentation result.

### Claim 14 (depends on 13)

The image processing device of claim 13, wherein the processor is arranged to execute the stored processor-executable instructions to perform operations of: before performing the image segmentation on the segmentation target according to the feature map through the target image segmentation model to obtain the target segmentation result,
obtaining a training sample image set, wherein the training sample image set comprises at least one training sample image corresponding to the segmentation target, and the training sample image has a labeled segmentation target;
performing the image segmentation on each training sample image through an initial target image segmentation model to obtain a predicted segmented image; and
performing the adversarial training of the initial target image segmentation model in combination with an initial image encoding model according to the labeled segmentation target and the predicted segmented image to obtain the target image segmentation model.

### Claim 15 (depends on 14)

The image processing device of claim 14, wherein performing the adversarial training of the initial target image segmentation model in combination with the initial image encoding model according to the labeled segmentation target and the predicted segmented image to obtain the target image segmentation model comprises:
encoding the labeled segmentation target and the predicted segmented image using the initial image encoding model to obtain a first space shape code and a second space shape code, wherein the first space shape code is a low-dimensional manifold space representation of the labeled segmentation target, and the second space shape code is a low-dimensional manifold space representation of the predicted segmented image;
calculating a code distance between the first space shape code and the second space shape code through a preset minimum square error loss function to obtain a prediction error; and
updating network parameters of the initial target image segmentation model based on a shape constraint of the prediction error, iterating the adversarial training process and terminating the adversarial training process until a final prediction error is less than a preset segmentation error threshold to obtain the target image segmentation model.

### Claim 16 (depends on 15)

The image processing device of claim 15, wherein the initial image encoding model comprises a decoding sub-model, and wherein the processor is arranged to execute the stored processor-executable instructions to perform operations of: after calculating the code distance between the first space shape code and the second space shape code through the preset minimum square error loss function to obtain the prediction error,
decoding and performing image reconstruction on the first space shape code and the second space shape code respectively through the decoding sub-model to obtain a first reconstructed image and a second reconstructed image; and
combining, through a preset reconstruction loss function, an error between the labeled segmentation target and the first reconstructed image with an error between the predicted segmented image and the second reconstructed image to obtain a reconstruction error of the initial image encoding model.

### Claim 17 (depends on 16)

The image processing device of claim 16, wherein updating the network parameters of the initial target image segmentation model based on the shape constraint of the prediction error, iterating the adversarial training process and terminating the adversarial training process until the final prediction error is less than the preset segmentation error threshold to obtain the target image segmentation model comprises:
updating the network parameters of the initial target image segmentation model using the prediction error in an M-th training, wherein M is an integer greater than or equal to 1;
obtaining a comprehensive encoding error based on the reconstruction error and the prediction error in an (M+1)-th training;
updating the network parameters of the initial image encoding model using the comprehensive encoding error; and
iterating the adversarial training process until the final prediction error is less than the preset encoding error threshold to obtain a target encoding model and the target image segmentation model.

### Claim 18 (depends on 15)

The image processing device of claim 15, wherein the initial image encoding model comprises an encoding sub-model comprising at least one convolution layer and a fully connected layer,
wherein encoding the labeled segmentation target and the predicted segmented image using the initial image encoding model to obtain the first space shape code and the second space shape code comprises:
squeezing the training sample image and the predicted segmented image through the at least one convolution layer according to a preset step size of the at least one convolution layer, and encoding the labeled segmentation target in a squeezed training sample image and a squeezed predicted segmented image through the fully connected layer to obtain the first space shape code and the second space shape code.

### Claim 19 (depends on 16)

The image processing device of claim 16, wherein the decoding sub-model comprises at least one de-convolution layer, a Rectified Linear Unit (ReLU) and a batch normalization layer.

### Claim 20

A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform an image processing method, the method comprising:
performing feature extraction on an image to be processed through a primary image segmentation model to obtain a feature map of the image to be processed; and
performing image segmentation on a segmentation target according to the feature map through a target image segmentation model to obtain a target segmentation result, wherein the target image segmentation model is obtained from an adversarial training with an image encoding model, and the image encoding model is used for placing a shape constraint on the target segmentation result.
