Method and apparatus for semantic segmentation and depth completion using a convolutional neural network

A computer-implemented method for semantic segmentation and/or depth completion of an inputted visible image and/or an inputted depth image, comprising:

• accessing a convolutional neural network (CNN) for semantic segmentation and/or depth completion of images;
• and
• performing semantic segmentation and/or depth completion of the inputted visible image and/or the inputted depth image using the CNN, comprising inputting a first subnetwork (S1) of the CNN with the inputted visible image and/or inputting a second subnetwork (S2) of the CNN with the inputted depth image,
• wherein the CNN is trained using a training set of image pairs, with each image pair in the training set having a visible image (xi) and a corresponding depth image (di), where at least one of the visible image (xi) and the corresponding depth image (di) is semantically segmented, by:
• (a) training the first subnetwork (S1) of the neural network, the first subnetwork (S1) being a first encoder-decoder network with:
• (i) an encoder branch that generates an outputted first feature map (xip) from a visible image (xi);
• and (ii) a decoder branch that generates a semantically segmented image ({circumflex over (x)}i) from an inputted first feature map (xir);
• wherein during training the encoder branch and the decoder branch of the first subnetwork (S1) are directly connected for setting the inputted first feature map (xir) as the outputted first feature map (xip);
• (b) training the second subnetwork (S2) of the neural network, the second subnetwork (S2) being a second encoder-decoder network with:
• (i) an encoder branch of the second subnetwork (S2) generating an outputted second feature map (dip) from a depth image (di);
• and (ii) a decoder branch of the second subnetwork (S2) either generating a semantically segmented image or a depth image ({circumflex over (d)}i) from an inputted second feature map (dir);
• wherein during training the encoder branch and the decoder branch of the second subnetwork (S2) are directly connected for setting the inputted second feature map (dir) as the outputted second feature map (dip);
• and
• (c) training a third subnetwork (S3) of the neural network arranged between the encoder branch and the decoder branch of the first subnetwork (S1) and the second subnetwork (S2) to reconstruct (i) the inputted first feature map (xir) from the outputted first feature map (xip) and (ii) the inputted second feature map (dir) from the outputted second feature map (dip);
• wherein third subnetwork S3 is trained to generate a common representation of the outputs reconstructed feature maps xr for visible images and dr for depth images, allowing either the view from the first subnetwork (S1) or the view from the second subnetwork (S2) to be reconstructed therefrom.

A computer-implemented method for generating a semantically segmented image and a depth completion image using a convolutional neural network (CNN) from an inputted visible image, comprising:

• accessing from a memory the CNN having a first subnetwork (S1), a second subnetwork (S2) and a third subnetwork (S3);
• and
• processing the inputted visible image using the first subnetwork (S1) of the CNN to produce a first runtime feature map (xip);
• processing the first runtime feature map (xip) using the third subnetwork (S3) of the CNN to produce a second runtime feature map (xir) and a third runtime feature map (dir) from a common representation;
• processing the second runtime feature map (xir) using the first subnetwork (S1) of the CNN to produce the semantically segmented image;
• processing the third runtime feature map (dir) using the second subnetwork (S2) of the CNN to produce the depth image;
• and
• outputting the semantically segmented image and the depth image;
• wherein the CCN is trained to produce the common representation using a training set of image pairs, with each image pair in the training set having a visible image (xi) and a corresponding depth image (di), where at least one of the visible image (xi) and the corresponding depth image (di) is semantically segmented.