Feature fusion for multi-modal machine learning analysis

A method of fusing features from a plurality of data modalities, comprising:

• for each data modality of the plurality of data modalities:
• determining one or more features for the data modality based on a sample dataset;
• training a first pre-trained model for the data modality based on the determined or more features and the sample dataset to produce a first pre-trained model output vector;
• training a first deep learning model for the data modality based on the sample dataset and one or more parameters of the first deep learning model to produce a deep learning model output vector, wherein the first deep learning model includes an inception-residual block to perform multi-scale feature map generation based on concatenation of feature maps;
• concatenating the deep learning model output vector and a second deep learning model output vector into a concatenated modality vector;
• submitting the concatenated modality vector to a first abstraction layer, wherein the first abstraction layer is to output a first abstraction layer output vector;
• concatenating the first abstraction layer output vector and the first pre-trained model output vector into a second concatenated layer;
• submitting the second concatenated layer to a second abstraction layer, the second abstraction layer to generate an output;
• comparing an error of the output with a threshold;
• updating the one or more parameters of the first deep learning model in response to the error of the output being above the threshold;
• partitioning audiovisual data received by a computer vision system into video frames and an audio signal;
• inputting the video frames into the first deep learning model and the audio signal into the first pre-trained model;
• and
• performing a computer vision-based emotion recognition task associated with the audiovisual data based on the first deep learning model, the first pre-trained model, and a test dataset.

A non-transitory machine readable storage medium comprising instructions to cause programmable circuitry to at least:

• for each data modality of a plurality of data modalities:
• determine one or more features for the data modality based on a sample dataset;
• train a first pre-trained model for the data modality based on the determined or more features and the sample dataset to produce a first pre-trained model output vector;
• train a first deep learning model for the data modality based on the sample dataset and one or more parameters of the first deep learning model to produce a deep learning model output vector, wherein the first deep learning model includes an inception-residual block to perform multi-scale feature map generation based on concatenation of feature maps;
• concatenate the deep learning model output vector and a second deep learning model output vector into a concatenated modality vector;
• submit the concatenated modality vector to a first abstraction layer, wherein the first abstraction layer is to output a first abstraction layer output vector;
• concatenate the first abstraction layer output vector and the first pre-trained model output vector into a second concatenated layer;
• submit the second concatenated layer to a second abstraction layer, the second abstraction layer to generate an output;
• compare an error of the output with a threshold;
• update the one or more parameters of the first deep learning model in response to the error of the output being above the threshold;
• partition audiovisual data received by a computer vision system into video frames and an audio signal;
• input the video frames into the first deep learning model and the audio signal into the first pre-trained model;
• and
• perform a computer vision-based emotion recognition task associated with the audiovisual data based on the first deep learning model, the first pre-trained model, and a test dataset.