Coupled multi-task fully convolutional networks using multi-scale contextual information and hierarchical hyper-features for semantic image segmentation

A computer-implemented method for performing semantic image segmentation comprising:

• implementing a multi-stage fully convolutional network based on an input image, wherein implementing the multi-stage fully convolutional network comprises:
• applying a first stage to output a first feature map at a first resolution;
• applying a second stage, subsequent to the first stage in the multi-stage fully convolutional network, to output a second feature map at a second resolution less than the first resolution;
• and
• applying a third stage, subsequent to the second stage in the multi-stage fully convolutional network, to output a third feature map at the second resolution;
• combining at least the second and third feature maps to generate a hyper-feature corresponding to the input image;
• up-sampling the hyper-feature to the first resolution;
• generating a final set of features corresponding to the input image by at least summing the up-sampled hyper-feature and the first feature map;
• and
• classifying the final set of features to provide a semantic image segmentation of the input image, the semantic image segmentation including pixel-level category labels for pixels of the input image.

A system for performing semantic image segmentation comprising:

• a memory storage configured to receive an input image;
• and
• a processor coupled to the memory storage, the processor to:
• implement a multi-stage fully convolutional network based on an input image, wherein to implement the multi-stage fully convolutional network, the processor is to apply a first stage to output a first feature map at a first resolution, to apply a second stage, subsequent to the first stage in the multi-stage fully convolutional network, to output a second feature map at a second resolution less than the first resolution, and to apply a third stage, subsequent to the second stage in the multi-stage fully convolutional network, to output a third feature map at the second resolution;
• combine at least the second and third feature maps to generate a hyper-feature corresponding to the input image;
• up-sample the hyper-feature to the first resolution;
• generate a final set of features corresponding to the input image by at least summing the up-sampled hyper-feature and the first feature map;
• and
• classify the final set of features to provide a semantic image segmentation of the input image, the semantic image segmentation including pixel-level category labels for pixels of the input image.

At least one machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform semantic image segmentation by:

• implementing a multi-stage fully convolutional network based on an input image, wherein implementing the multi-stage fully convolutional network comprises:
• applying a first stage to output a first feature map at a first resolution;
• applying a second stage, subsequent to the first stage in the multi-stage fully convolutional network, to output a second feature map at a second resolution less than the first resolution;
• and
• applying a third stage, subsequent to the second stage in the multi-stage fully convolutional network, to output a third feature map at the second resolution;
• combining at least the second and third feature maps to generate a hyper-feature corresponding to the input image;
• up-sampling the hyper-feature to the first resolution;
• generating a final set of features corresponding to the input image by at least summing the up-sampled hyper-feature and the first feature map;
• and
• classifying the final set of features to provide a semantic image segmentation of the input image, the semantic image segmentation including pixel-level category labels for pixels of the input image.