Training method and apparatus for human-factor intelligence state monitoring model, and human-factor intelligence state monitoring method and apparatus

A human-factor intelligent state monitoring method with multimodal synchronous data fusion, wherein the human-factor intelligent state monitoring method comprises:

• collecting physiological signals and a video signal of a person, the physiological signals comprising an electrocardiographic signal, an electroencephalographic signal, and an electromyographic signal;
• and
• inputting the physiological signals and the video signal into the human-factor intelligent state monitoring model with multimodal synchronous data fusion obtained by a training method for the human-factor intelligent state monitoring model with multimodal synchronous data fusion to generate a corresponding prediction result, to determine a human-factor intelligent state,
• wherein the training method comprises:
• constructing a training sample set, the training sample set comprising a plurality of samples, each of the plurality of samples comprising collected physiological signals and a video signal of a person, and the physiological signals comprising an electrocardiographic signal, an electroencephalographic signal, and an electromyographic signal;
• and adding a real label to each of the plurality of samples;
• constructing an initial neural network model, the initial neural network model comprising a data preprocessing module, a data fusion module, a public feature extraction module, a multi-head feature extraction module, and a prediction output layer, wherein:
• the public feature extraction module comprises a convolution layer, a dense fusion layer, and a spatial pyramid pooling layer;
• the data preprocessing module is configured to perform first data preprocessing operation on the physiological signals in each sample to obtain two-dimensional physiological signals, extract a key frame of the video signal in each sample by using a predetermined deep learning method, and perform second data preprocessing operation on the key frame to obtain a two-dimensional video signal;
• the data fusion module is configured to fuse the two-dimensional physiological signals and the two-dimensional video signal in a channel direction to generate a three-dimensional fusion signal;
• the public feature extraction module is configured to perform feature extraction and fusion on the three-dimensional fusion signal by using the convolution layer and the dense fusion layer to generate feature maps of different scales, and summarize the feature maps through pyramid pooling operations of different scales by using the spatial pyramid pooling layer;
• the multi-head feature extraction module is configured to fuse the feature maps of different scales obtained by the public feature extraction module to obtain a high-dimensional feature map;
• and
• the prediction output layer is configured to generate a prediction result of a corresponding sample according to the high-dimensional feature map;
• and
• training the initial neural network model by using the training sample set, constructing a loss between the prediction result and the real label, adjusting a parameter of the initial neural network model by using the loss, and finally obtaining the human-factor intelligent state monitoring model with multimodal synchronous data fusion.

A device, comprising:

• a processor;
• and
• a memory configured to store a computer program, wherein the processor is configured to invoke and run the computer program stored in the memory to execute a training method for the human-factor intelligent state monitoring model with multimodal synchronous data fusion,
• wherein the training method comprises:
• constructing a training sample set, the training sample set comprising a plurality of samples, each of the plurality of samples comprising collected physiological signals and a video signal of a person, and the physiological signals comprising an electrocardiographic signal, an electroencephalographic signal, and an electromyographic signal;
• and adding a real label to each of the plurality of samples;
• constructing an initial neural network model, the initial neural network model comprising a data preprocessing module, a data fusion module, a public feature extraction module, a multi-head feature extraction module, and a prediction output layer, wherein:
• the public feature extraction module comprises a convolution layer, a dense fusion layer, and a spatial pyramid pooling layer;
• the data preprocessing module is configured to perform first data preprocessing operation on the physiological signals in each sample to obtain two-dimensional physiological signals, extract a key frame of the video signal in each sample by using a predetermined deep learning method, and perform second data preprocessing operation on the key frame to obtain a two-dimensional video signal;
• the data fusion module is configured to fuse the two-dimensional physiological signals and the two-dimensional video signal in a channel direction to generate a three-dimensional fusion signal;
• the public feature extraction module is configured to perform feature extraction and fusion on the three-dimensional fusion signal by using the convolution layer and the dense fusion layer to generate feature maps of different scales, and summarize the feature maps through pyramid pooling operations of different scales by using the spatial pyramid pooling layer;
• the multi-head feature extraction module is configured to fuse the feature maps of different scales obtained by the public feature extraction module to obtain a high-dimensional feature map;
• and
• the prediction output layer is configured to generate a prediction result of a corresponding sample according to the high-dimensional feature map;
• and
• training the initial neural network model by using the training sample set, constructing a loss between the prediction result and the real label, adjusting a parameter of the initial neural network model by using the loss, and finally obtaining the human-factor intelligent state monitoring model with multimodal synchronous data fusion.