Apparatus for estimating emotion using multimodal model and method of training the same

An apparatus for estimating emotion, the apparatus comprising:

• a speech emotion representation (SER) model processing unit configured to input an acquired speaker identity feature and an acquired speaker speech data feature to a predetermined SER model, encode the speaker identity feature and the speaker speech data feature through a speech encoder in the SER model, and encode the encoded speaker identity feature and speaker speech data feature through a disentangled feature encoder to select data related to emotion and exclude data unrelated to emotion from the encoded speaker identity feature and the encoded speaker speech data feature, and output a phonetic feature query, a phonetic feature key, and a phonetic feature value of a speaker;
• a text emotion representation (TER) model processing unit configured to input a text feature acquired from transferred text corresponding to the speaker identity feature and the speaker speech data feature to a predetermined TER model, and encode the text feature through a text encoder in the TER model to output a text feature query, a text feature key, and a text feature value of the text feature;
• and
• a representation fusion (RF) model processing unit configured to:
• input the output phonetic feature query, phonetic feature key, and phonetic feature value of the speaker and the output text feature query, text feature key, and text feature value to a predetermined RF model;
• apply a first cross-attention mechanism to the output phonetic feature query, text feature key, and text feature value;
• apply a second cross-attention mechanism to the output text feature query, phonetic feature key, and phonetic feature value;
• apply a first Add and Normalization layer, a first feedforward layer, a second Add and Normalization layer, and a first max pooling layer to an output of the first cross-attention mechanism;
• apply a third Add and Normalization layer, a second feedforward layer, a fourth Add and Normalization layer, and a second max pooling layer to an output of the second cross-attention mechanism;
• and
• output a probability for each emotion category of the speaker based on outputs of the first and second max pooling layers.