Learning method and learning device for improving image segmentation and testing method and testing device using the same

A testing method for performing image segmentation on a test image as an input image by using a computing device having a processor and including a convolutional neural network, comprising steps of:

• (a) a testing device receiving or supporting another device to receive the test image, on conditions that (I) the computing device includes (i) an encoding layer having each of a (1-1)-th to a (1-K)-th filters respectively generating a (1-1)-th to a (1-K)-th feature maps for training by applying one or more convolution operations to a training image;
• (ii) a decoding layer having each of a (2-K)-th to a (2-1)-th filters respectively generating a (2-K)-th to a (2-1)-th feature maps for training by applying one or more deconvolution operations to the (1-K)-th feature maps for training;
• and (iii) an intermediate layer having each of a (3-1)-th to a (3-H)-th filters respectively arranged between each of H encoding filters among the K filters included in the encoding layer and each of H decoding filters among the K filters included in the decoding layer, (II) the computing device performs processes of (i) acquiring, if the training image is obtained, the (1-1)-th to the (1-K)-th feature maps for training through the encoding layer;
• (ii) acquiring a (3-1)-th to a (3-H)-th feature maps for training by respectively inputting each output of the H encoding filters to the (3-1)-th to the (3-H)-th filters;
• (iii) sequentially acquiring the (2-K)-th to the (2-1)-th feature maps for training from the decoding layer either by the respective H decoding filters respectively using both the (3-1)-th to the (3-H)-th feature maps for training and feature maps for training obtained from respective previous decoding filters of the respective H decoding filters or by respective K-H decoding filters that are not associated with the (3-1)-th to the (3-H)-th filters using feature maps for training gained from respective previous decoding filters of the respective K-H decoding filters;
• and (iv) adjusting parameters of at least part of the (2-1)-th to the (2-K)-th filters and the (3-1)-th to the (3-H)-th filters and the (1-1)-th to the (1-K)-th filters by performing a backpropagation process with a difference between a Ground Truth (GT) label image and the (2-1)-th feature maps for training;
• (b) the testing device acquiring or supporting another device to acquire the (1-1)-th to the (1-K)-th feature maps for testing through the encoding layer;
• (c) the testing device acquiring or supporting another device to acquire a (3-1)-th to a (3-H)-th feature maps for testing by respectively inputting each output of the H encoding filters to the (3-1)-th to (3-H)-th filters;
• (d) the testing device performing a process of sequentially acquiring the (2-K)-th to the (2-1)-th feature maps for testing either by (i) the respective H decoding filters respectively using both the (3-1)-th to the (3-H)-th feature maps for testing and feature maps for testing obtained from respective previous decoding filters of the respective H decoding filters or by (ii) respective K-H decoding filters that are not associated with the (3-1)-th to the (3-H)-th filters using feature maps for testing gained from respective previous decoding filters of the respective K-H decoding filters;
• and
• (e) the testing device outputting a label image from the decoding layer based on the (1-1)-th to a (1-K)-th feature maps, the (2-K)-th to a (2-1)-th feature maps, and the (3-1)-th to the (3-H)-th feature maps,
• wherein the label image is a set of the semantic segments with clear boundaries such that the semantic segments collectively cover the test image.

A computing device including a convolutional neural network for improving image segmentation, wherein the computing device includes (i) an encoding layer having each of a (1-1)-th to a (1-K)-th filters respectively generating a (1-1)-th to a (1-K)-th feature maps by applying one or more convolution operations to a training image as an input image;

• (ii) a decoding layer having each of a (2-K)-th to a (2-1)-th filters respectively generating a (2-K)-th to a (2-1)-th feature maps by applying one or more deconvolution operations to the (1-K)-th feature maps;
• and (iii) an intermediate layer having each of a (3-1)-th to a (3-H)-th filters respectively arranged between each of H encoding filters among the K filters included in the encoding layer and each of H decoding filters among the K filters included in the decoding layer, comprising:
• a communication part that receives the input image;
• a processor that performs processes of (I) acquiring, if the input image is obtained, the (1-1)-th to the (1-K)-th feature maps through the encoding layer;
• (II) acquiring a (3-1)-th to a (3-H)-th feature maps by respectively inputting each output of the H encoding filters to the (3-1)-th to the (3-H)-th filters;
• (Ill) sequentially acquiring the (2-K)-th to the (2-1)-th feature maps either by (i) the respective H decoding filters respectively using both the (3-1)-th to the (3-H)-th feature maps and feature maps obtained from respective previous decoding filters of the respective H decoding filters or by (ii) respective K-H decoding filters that are not associated with the (3-1)-th to the (3-H)-th filters using feature maps gained from respective previous decoding filters of the respective K-H decoding filters;
• and (IV) adjusting parameters of at least part of the (2-1)-th to the (2-K)-th filters and the (3-1)-th to the (3-H)-th filters and the (1-1)-th to the (1-K)-th filters by performing a backpropagation process with a difference between a Ground Truth (GT) label image and the (2-1)-th feature map;
• and
• wherein the processor outputs a label image from the decoding layer based on the (1-1)-th to a (1-K)-th feature maps, the (2-K)-th to a (2-1)-th feature maps, and the (3-1)-th to the (3-H)-th feature maps,
• wherein the label image is a set of the semantic segments with clear boundaries such that the semantic segments collectively cover the test image.

A testing device including a convolutional neural network for performing image segmentation on a test image as an input image by using a computing device, comprising:

• a communication part that receives the test image, on conditions that (I) the computing device includes (i) an encoding layer having each of a (1-1)-th to a (1-K)-th filters respectively generating a (1-1)-th to a (1-K)-th feature maps for training by applying one or more convolution operations to a training image;
• (ii) a decoding layer having each of a (2-K)-th to a (2-1)-th filters respectively generating a (2-K)-th to a (2-1)-th feature maps for training by applying one or more deconvolution operations to the (1-K)-th feature maps for training;
• and (iii) an intermediate layer having each of a (3-1)-th to a (3-H)-th filters respectively arranged between each of H encoding filters among the K filters included in the encoding layer and each of H decoding filters among the K filters included in the decoding layer, (II) the computing device performs processes of (i) acquiring, if the training image is obtained, the (1-1)-th to the (1-K)-th feature maps for training through the encoding layer;
• (ii) acquiring a (3-1)-th to a (3-H)-th feature maps for training by respectively inputting each output of the H encoding filters to the (3-1)-th to the (3-H)-th filters;
• (iii) sequentially acquiring the (2-K)-th to the (2-1)-th feature maps for training from the decoding layer either by the respective H decoding filters respectively using both the (3-1)-th to the (3-H)-th feature maps for training and feature maps for training obtained from respective previous decoding filters of the respective H decoding filters or by respective K-H decoding filters that are not associated with the (3-1)-th to the (3-H)-th filters using feature maps for training gained from respective previous decoding filters of the respective K-H decoding filters;
• and
• (iv) adjusting parameters of at least part of the (2-1)-th to the (2-K)-th filters and the (3-1)-th to the (3-H)-th filters and the (1-1)-th to the (1-K)-th filters by performing a backpropagation process with a difference between a Ground Truth (GT) label image and the (2-1)-th feature maps for training;
• a processor that performs processes of (I) acquiring the (1-1)-th to the (1-K)-th feature maps for testing through the encoding layer;
• (II) acquiring a (3-1)-th to a (3-H)-th feature maps for testing by respectively inputting each output of the H encoding filters to the (3-1)-th to (3-H)-th filters;
• and (III) the testing device performing a process of sequentially acquiring the (2-K)-th to the (2-1)-th feature maps for testing either by (i) the respective H decoding filters respectively using both the (3-1)-th to the (3-H)-th feature maps for testing and feature maps for testing obtained from respective previous decoding filters of the respective H decoding filters or by (ii) respective K-H decoding filters that are not associated with the (3-1)-th to the (3-H)-th filters using feature maps for testing gained from respective previous decoding filters of the respective K-H decoding filters;
• and
• wherein the processor outputs a label image from the decoding layer based on the (1-1)-th to a (1-K)-th feature maps, the (2-K)-th to a (2-1)-th feature maps, and the (3-1)-th to the (3-H)-th feature maps,
• wherein the label image is a set of the semantic segments with clear boundaries such that the semantic segments collectively cover the test image.