Method and system for multimodal analysis based emotion recognition

A real time multimodal assisting system for multimodal analysis for emotion recognition, the system comprising:

• a processor;
• and
• a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which on execution, cause the processor to:
• segment video data of a user into a plurality of video segments, wherein each of the plurality of video segments are identified by a video segment identifier;
• extract a plurality of visual features, a plurality of voice features and a plurality of text features from the plurality of video segments, wherein the plurality of visual features, the plurality of voice features, and the plurality of text features are identified by the corresponding video segment identifier, wherein the text features are extracted from voice data in the video data of a user;
• determine autocorrelation values among each of the plurality of visual features, the plurality of voice features, and the plurality of text features;
• align each of the plurality of visual features, the plurality of voice features and the plurality of text features based on the video segment identifier and the autocorrelation values to obtain a plurality of aligned multimodal features;
• determine one of two classes of emotions for each of the plurality of aligned multimodal features based on correlation values determined for each of the plurality of aligned multimodal features, wherein the two classes of emotions comprise positive class of emotions, and negative class of emotions;
• compare the determined emotion for each of the plurality of aligned multimodal features with historic multimodal features from a database;
• and
• determine emotion of the user at real time based on comparison.

A non-transitory computer readable medium including instructions stored thereon that when processed by at least one processor cause a multimodal assisting system to perform operations comprising:

• segmenting video data of a user into a plurality of video segments, wherein each of the plurality of video segments are identified by a video segment identifier;
• extracting a plurality of visual features, a plurality of voice features and a plurality of text features from the plurality of video segments, wherein the plurality of visual features, the plurality of voice features, and the plurality of text features are identified by the corresponding video segment identifier, wherein the text features are extracted from voice data in the video data of a user;
• determining autocorrelation values among each of the plurality of visual features, the plurality of voice features, and the plurality of text features;
• aligning each of the plurality of visual features, the plurality of voice features and the plurality of text features based on the video segment identifier and the autocorrelation values to obtain a plurality of aligned multimodal features;
• determining one of two classes of emotions for each of the plurality of aligned multimodal features based on correlation values determined for each of the plurality of aligned multimodal features, wherein the two classes of emotions comprise positive class of emotions, and negative class of emotions;
• comparing the determined emotion for each of the plurality of aligned multimodal features with historic multimodal features from a database;
• and
• determining emotion of the user at real time based on comparison.