# US 2021/0241107 A1 · Method, apparatus, device, and storage medium for training image semantic segmentation network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241107 A1 - Document ID: US20210241107A1 - Status: Published application - Kind: A1 - Application number: 17/235,046 - Filed: 2021-04-20 - Published: 2021-08-05 - Earliest claimed: 2019-03-26 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Ze Qun Jie; Wei Liu - Web page: https://patentgrep.com/patent/US20210241107A1 ## CPC - G06N 3/08: Learning methods - G06T 7/11: Region-based segmentation - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - Y02T 10/40: Engine management systems - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION ## Abstract A method for training an image semantic segmentation network is provided. The method includes: training a first image semantic segmentation network based on a training image set, training images in the training image set including annotation information; performing mask processing on network weights of the first image semantic segmentation network by using a mask network, to obtain a second image semantic segmentation network, the mask network being configured to select network weights in the first image semantic segmentation network that are insensitive to a distribution of image features in the training image set; and training the second image semantic segmentation network based on the training image set and a test image set, the distribution of image features in the training image set being different from a distribution of image features in the test image set. ## Claims ### Claim 1 A method for training an image semantic segmentation network, executed by a computer device, the method comprising: training a first image semantic segmentation network based on a training image set, training images in the training image set comprising annotation information; performing mask processing on network weights of the first image semantic segmentation network by using a mask network, to obtain a second image semantic segmentation network, the mask network being configured to select network weights in the first image semantic segmentation network that are insensitive to a distribution of image features in the training image set; and training the second image semantic segmentation network based on the training image set and a test image set, test images in the test image set comprising no annotation information, and the distribution of image features in the training image set being different from a distribution of image features in the test image set. ### Claim 2 (depends on 1) The method according to claim 1, wherein a network structure of the mask network is the same as a network structure of the first image semantic segmentation network, and a number of network weights in the mask network is the same as a number of network weights in the first image semantic segmentation network; and the performing the mask processing comprises: obtaining a first weight array corresponding to the first image semantic segmentation network, the first weight array comprising network weights corresponding to convolution kernels in the first image semantic segmentation network; performing, by using a real-valued mask corresponding to the mask network, mask processing on the first weight array to obtain a second weight array, the real-valued mask comprising network weights corresponding to convolution kernels in the mask network; and generating the second image semantic segmentation network based on the second weight array. ### Claim 3 (depends on 2) The method according to claim 2, wherein the performing the mask processing on the first weight array comprises: performing, by using a threshold function, binarization processing on the real-valued mask to generate a binary mask, the binary mask being an array of binary values of ‘0’ or ‘1’, and a size of the binary mask being the same as a size of the first weight array; and performing pointwise multiplication on the first weight array and the binary mask to obtain the second weight array. ### Claim 4 (depends on 3) The method according to claim 3, the method further comprising: performing the pointwise multiplication on the first weight array and the binary mask to obtain an intermediate weight array; and generating the second weight array based on the intermediate weight array and a perturbation array, the perturbation array being configured to increase perturbation to network weights in the intermediate weight array. ### Claim 5 (depends on 2) The method according to claim 2, wherein the generating the second image semantic segmentation network comprises: updating at least one network weight on each convolution kernel in the first image semantic segmentation network based on the second weight array; and determining, as the second image semantic segmentation network, the first image semantic segmentation network obtained after updating the at least one network weight. ### Claim 6 (depends on 1) The method according to claim 1, wherein the training the second image semantic segmentation network comprises: obtaining a segmentation loss of the second image semantic segmentation network based on the training image set; obtaining a discriminator loss of a discriminator based on the training image set and the test image set, the discriminator being configured to determine, based on an image feature of an image, whether the image belongs to the training image set or the test image set; and performing adversarial training on the second image semantic segmentation network and the discriminator based on the segmentation loss and the discriminator loss. ### Claim 7 (depends on 6) The method according to claim 6, wherein the obtaining the discriminator loss comprises: inputting a target image to the second image semantic segmentation network to obtain a target image feature, the target image belonging to the training image set or the test image set; inputting the target image feature to the discriminator to obtain a discrimination result; and obtaining the discriminator loss based on an image set to which the target image belongs and the discrimination result. ### Claim 8 (depends on 1) The method according to claim 1, wherein the distribution of image features in the training image set is determined to be different from a distribution of image features in the test image set based on at least one of a difference, between the training images and the test images, in a place or a time of image acquisition, in a characteristic of an object included in an image, or in an image acquisition environment. ### Claim 9 (depends on 1) An image semantic segmentation method, applicable to a computer device equipped with a target image semantic segmentation network, the target image semantic segmentation network being trained using the method according to claim 1, the image semantic segmentation method comprising: obtaining a test image in a second test image set; inputting the test image to the target image semantic segmentation network, and obtaining a target image semantic segmentation result outputted by the target image semantic segmentation network, the target image semantic segmentation result comprising categories corresponding to objects to which pixels in the test image belong; and displaying, based on the target image semantic segmentation result, a segmented image corresponding to the test image, different categories of objects being annotated in the segmented image. ### Claim 10 An apparatus for training an image semantic segmentation network, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: first training code configured to cause at least one of the at least one processor to train a first image semantic segmentation network based on a training image set, training images in the training image set comprising annotation information; masking code configured to cause at least one of the at least one processor to perform mask processing on network weights of the first image semantic segmentation network through a mask network, to obtain a second image semantic segmentation network, the mask network being configured to select network weights in the first image semantic segmentation network that are insensitive to a distribution of image features in the training image set; and second training code configured to cause at least one of the at least one processor to train the second image semantic segmentation network based on the training image set and a test image set, test images in the test image set comprising no annotation information, and the distribution of image features in the training image set being different from a distribution of image features in the test image set. ### Claim 11 (depends on 10) The apparatus according to claim 10, wherein a network structure of the mask network is the same as a network structure of the first image semantic segmentation network, and a number of network weights in the mask network is the same as a number of network weights in the first image semantic segmentation network; and the masking code comprises: obtaining subcode configured to cause at least one of the at least one processor to obtain a first weight array corresponding to the first image semantic segmentation network, the first weight array comprising network weights corresponding to convolution kernels in the first image semantic segmentation network; masking subcode configured to cause at least one of the at least one processor to perform mask processing on the first weight array through a real-valued mask corresponding to the mask network, to obtain a second weight array, the real-valued mask comprising network weights corresponding to convolution kernels in the mask network; and generation subcode configured to cause at least one of the at least one processor to generate the second image semantic segmentation network based on the second weight array. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein the masking subcode is further configured to cause at least one of the at least one processor to: perform binarization processing on the real-valued mask through a threshold function, to generate a binary mask, the binary mask being an array of binary values of ‘0’ or ‘1’, and a size of the binary mask being the same as a size of the first weight array; and perform pointwise multiplication on the first weight array and the binary mask to obtain the second weight array. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein the masking subcode is further configured to cause at least one of the at least one processor to: perform the pointwise multiplication on the first weight array and the binary mask to obtain an intermediate weight array; and generate the second weight array based on the intermediate weight array and a perturbation array, the perturbation array being configured to increase perturbation to network weights in the intermediate weight array. ### Claim 14 (depends on 11) The apparatus according to claim 11, wherein the generation subcode is further configured to cause at least one of the at least one processor to: update at least one network weight on each convolution kernel in the first image semantic segmentation network based on the second weight array; and determine, as the second image semantic segmentation network, the first image semantic segmentation network obtained after updating the at least one network weight. ### Claim 15 (depends on 10) The apparatus according to claim 10, wherein the second training code comprises: segmentation loss obtaining subcode configured to cause at least one of the at least one processor to obtain a segmentation loss of the second image semantic segmentation network based on the training image set; discriminator loss obtaining subcode configured to cause at least one of the at least one processor to obtain a discriminator loss of a discriminator based on the training image set and the test image set, the discriminator being configured to determine, based on an image feature of an image, whether the image belongs to the training image set or the test image set; and adversarial training subcode configured to cause at least one of the at least one processor to obtain perform adversarial training on the second image semantic segmentation network and the discriminator based on the segmentation loss and the discriminator loss. ### Claim 16 (depends on 15) The apparatus according to claim 15 wherein the discriminator loss obtaining subcode is further configured to cause at least one of the at least one processor to: input a target image to the second image semantic segmentation network to obtain a target image feature, the target image belonging to the training image set or the test image set; input the target image feature to the discriminator to obtain a discrimination result; and obtain the discriminator loss based on an image set to which the target image belongs and the discrimination result. ### Claim 17 (depends on 10) The apparatus according to claim 10, wherein the program code further comprises: determination code configured to cause at least one of the at least one processor to determine that the distribution of image features in the training image set is different from a distribution of image features in the test image set based on at least one of a difference, between the training images and the test images, in a place or a time of image acquisition, in a characteristic of an object included in an image, or in an image acquisition environment. ### Claim 18 (depends on 10) The apparatus according to claim 10, wherein the program code further comprises: target result obtaining code configured to cause at least one of the at least one processor to input an image to the second image semantic segmentation network, and obtain a target image semantic segmentation result outputted by the second image semantic segmentation network, the target image semantic segmentation result comprising categories corresponding to objects to which pixels in the image belong; and display code configured to cause at least one of the at least one processor to display, based on the target image semantic segmentation result, a segmented image corresponding to the image, different categories of objects being annotated in the segmented image. ### Claim 19 (depends on 1) A computer device, comprising a memory and a processor, the memory storing computer-readable instructions, the computer-readable instructions, when executed by the processor, causing the processor to perform operations of the method according to claim 1. ### Claim 20 A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors, causing at least one of the one or more processors to perform: training a first image semantic segmentation network based on a training image set, training images in the training image set comprising annotation information; performing mask processing on network weights of the first image semantic segmentation network by using a mask network, to obtain a second image semantic segmentation network, the mask network being configured to select network weights in the first image semantic segmentation network that are insensitive to a distribution of image features in the training image set; and training the second image semantic segmentation network based on the training image set and a test image set, test images in the test image set comprising no annotation information, and the distribution of image features in the training image set being different from a distribution of image features in the test image set.