Artificial intelligence device for light transformer-based emotion recognition (lter) and method thereof

A method for controlling an artificial intelligence (AI) deice, the method comprising:

• receiving, by a processor in the AI device, a video segment including a plurality of frames and an audio signal corresponding to the video segment;
• processing the audio signal, by an audio encoder, to generate an audio embedding;
• processing the video segment, by a visual encoder, to generate a visual embedding;
• processing the audio embedding, by an audio transformer, to generate an audio feature vector, a key matrix and a value matrix;
• processing the visual embedding, by a visual transformer, to generate a visual feature vector, wherein the processing the visual embedding includes performing cross-attention based on the key matrix and the value matrix from the audio transformer;
• generating a fused output using a fusion module that combines at least the audio feature vector and the visual feature vector;
• and
• generating an emotion prediction using a classifier module that analyzes the fused output, and outputting the emotion prediction.