Multimodal emotion recognition brain-computer interface instrument based on contrastive learning gated network

A multimodal emotion recognition instrument based on a contrastive learning gated network, wherein the instrument comprises an electroencephalogram (EEG) data collection apparatus configured to collect EEG data, an eye movement data collection apparatus configured to collect eye movement data, and a multimodal data processing apparatus;

• the multimodal data processing apparatus comprises a microprogrammed control unit (MCU), a neural network accelerator, a data bus, a data memory, and an output interface;
• and the MCU controls the neural network accelerator to acquire EEG data and eye movement data from the data memory to train the multimodal contrastive learning gated network, specifically comprising:
• a unimodal feature encoder pre-training module, configured to respectively input EEG primary features and eye movement primary features into an EEG feature encoder and an eye movement feature encoder, and respectively input encoding results into a classifier to respectively predict unimodal emotion categories;
• a multimodal contrastive learning training module, configured to respectively input EEG primary features and eye movement primary features into feature encoders of the unimodal feature encoder pre-training module to obtain multimodal features EEG and Eye, then respectively use an EEG similarity mapper and an EEG dissimilarity mapper as well as an eye similarity mapper and an eye dissimilarity mapper to decompose EEG and eye movement signals, fuse decomposition results, and perform intra-sample contrastive learning and inter-sample contrastive learning;
• and
• a gated structure training module, configured to fuse EEG primary features and eye movement primary features and then input the fused EEG primary features and eye movement primary features into a gated network, the gated network generating a two-dimensional vector as an output to select and activate different expert network branches to complete emotion classification tasks.