Fabric defect detection method based on multi-modal deep learning

A fabric defect detection method based on multi-modal deep learning, wherein the fabric defect detection method comprises:

• step 1:
• establishing a fabric detection data set based on different types of defects, comprising:
• placing a tactile sensor onto surfaces of fabrics with different fabric defects to collect fabric texture images, the different fabric defects comprises normal, structural defects and color defects, the structural defects comprise scraping, thinning, neps, holes, rovings, creases and running stitches, and the color defects comprise dirt, skilteriness, colored yarns, uneven dyes, black dots, skip printing and dark stripes;
• using an external camera to collect fabric external images of the fabrics at same positions and same angles where the tactile sensor collects the fabric texture images, taking each of the fabric external images and a corresponding one fabric texture image of the fabric texture images as a set of fabric detection data to thereby obtain sets of fabric detection data, and taking a corresponding fabric defect of the different fabric defects as a defect label for the fabric external image and the corresponding one fabric texture image, and each set of the sets of fabric detection data comprises one fabric texture image, one fabric external image and one defect label;
• and
• taking the sets of fabric detection data as the fabric detection data set;
• step 2:
• establishing a classification model based on multi-modal deep learning, comprising:
• connecting a feature extraction network and a multi-modal fusion network to establish the classification model based on multi-modal deep learning, wherein the feature extraction network comprises two parallel ResNet-50 networks, the two parallel ResNet-50 networks are configured to:
• respectively use the one fabric texture image and the one fabric external image in each set of the sets of fabric detection data as inputs to extract features of the one fabric texture image and one fabric external image to thereby obtain two vectors each with a length of N, where “N” is defined, and connect the two vectors to obtain a vector with a length of 2N as an extracted feature vector output;
• the multi-modal fusion network adopts 2 to 4 layers of fully connected networks to detect fabric defects;
• an output of a previous layer of the 2 to 4 layers of fully connected networks is used as an input of a subsequent layer of the 2 to 4 layers of fully connected networks, an input of a first layer of the 2 to 4 layers of fully connected networks is the extracted feature vector output with the length of 2N obtained by the feature extraction network, an output of a last layer of the 2 to 4 layers of fully connected networks is a feature vector characterizing the fabric defects, a length of the feature vector characterizing the fabric defects is equal to a type number of the one defect label contained in the each set of the sets of fabric detection data, and each element of the feature vector characterizing the fabric defects represents a probability of each of the fabric defects;
• step 3:
• training a fabric defect detection model, comprising:
• classifying the fabric detection data set obtained in the step 1 into a training set and a test set;
• inputting a fabric texture image and a fabric external image under a same set of fabric detection data in the training set into the two parallel ResNet-50 networks of the classification model established in the step 2, respectively;
• using a back propagation algorithm to train the fabric defect detection model;
• using a loss function Softmax Loss to constrain a training process of the fabric defect detection model;
• using the test set to determine a training effect of the fabric defect detection model to thereby obtain a trained fabric defect detection model;
• and
• step 4:
• collecting fabric texture images of a fabric to be detected and corresponding fabric external images of the fabric to be detected, inputting the fabric texture images of the fabric to be detected and the corresponding fabric external images of the fabric to be detected into the trained fabric defect detection model to detect a fabric defect of the fabric to be detected, wherein defect is a highest-confidence defect label in a feature vector characterizing fabric defects that is output by the trained fabric defect detection model.