Use of HCNN to predict lane lines types

A system for predicting lane line types utilizing a heterogeneous convolutional neural network (HCNN), the system comprising:

• one or more optical sensors disposed on a host member, the one or more optical sensors capturing an input image;
• and
• an HCNN having at least three distinct sub-networks, the three distinct sub-networks comprising:
• a first sub-network predicting object locations in the input image;
• a second sub-network predicting lane line locations in the input image;
• and
• a third sub-network predicting lane line types for each predicted lane line in the input image, and
• wherein the HCNN receives the input image and passes the input image through the three distinct sub-networks, wherein when the HCNN passes the input image through the three distinct sub-networks, the input image is directly received in a feature extraction layer (FEL) portion of the HCNN, the HCNN having multiple convolution, pooling and activation layers stacked together with each other;
• the FEL portion conducts a learning operation to learn to represent at least a first stage of data of the input image in a form including horizontal and vertical lines and simple blobs of colors, and outputting the first stage of data to at least:
• the first sub-network, the second sub-network, and the third sub-network;
• the first sub-network directly receives the data from the FEL portion and performs a first task of object detection, classification, and localization for classes of objects in the input image to create a detected object table;
• the second sub-network directly receives the data from the FEL portion and performs a second task of lane line detection to create a lane line location table;
• and
• the third sub-network directly receives the data from the FEL portion and performs a third task of lane line type detection to create a lane line type table.

A method of predicting lane line types utilizing a heterogeneous convolutional neural network (HCNN), the method comprising:

• capturing an input image with one or more optical sensors disposed on a host member;
• passing the input image through four shared convolution and pooling convolutional layer portions (CPLs) within the HCNN, the HCNN having at least three distinct sub-networks;
• and
• passing an output of the four shared convolutional layers through at least three fully connected layers within the HCNN;
• training a first sub-network by minimizing a loss function of the first sub-network while freezing a second sub-network and a third sub-network;
• training the second sub-network by minimizing a loss function of the second sub-network while freezing the first sub-network and the third sub-network;
• and
• training the third sub-network by minimizing a loss function of the third sub-network while freezing the first sub-network and the second sub-network, and
• wherein the first sub-network performs a first task of object detection, classification, and localization for classes of objects in the input image to create a detected object table, the second sub-network performs a second task of lane line detection and localization for classes of lane lines in the input image to create a lane line location table, and the third sub-network performs a third task of lane line type detection for classes of lane lines types in the input image and creates a detected lane line type table.