Coal gangue recognition method based on visible-near-infrared spectrum and image multi-modal information fusion

A coal gangue recognition method based on visible-near-infrared spectrum and image multi-modal information fusion, comprising:

• S1:
• collecting spectral information and image information about a sample to be recognized;
• S2:
• preprocessing the spectral information and the image information respectively, and constructing an image-spectrum data set with spectra and images having a one-to-one correspondence relationship to acquire model priori knowledge, the image-spectrum data set comprising a spectral data set and an image data set;
• S3:
• extracting spectral features from the spectral data set by using a spectral feature extraction neural network model;
• and extracting image features from the image data set by using an image feature extraction neural network model;
• S4:
• inputting the spectral features and the image features obtained from feature extraction into a two-stream fusion network to output a comprehensive feature;
• S5:
• inputting extracted spectral features, extracted image features and the comprehensive feature into a spectral branch classifier, an image branch classifier and a fusion branch classifier, respectively, to obtain class confidence under three classifiers;
• S6:
• calculating importance weights of a spectral branch, an image branch and an image-spectrum fusion branch;
• and
• S7:
• subjecting the importance weights and corresponding confidence to multiply-accumulate operation to obtain a score matrix of coal gangue, and using the score matrix to achieve coal gangue recognition.