Multi-modal cross attention sentiment analysis of textual and audio embeddings

A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing sentiment analysis, the method comprising:

• obtaining, by a data classification system, raw data, associated with a meeting, wherein the raw data the comprises a textual transcript and an audio file of the meeting;
• applying a generative adversarial network (GAN) to the audio file to obtain audio embeddings and extracted text;
• applying the textual transcripts and the extracted text to a trained language model to obtain textual embeddings;
• applying the audio embeddings and the textual embeddings to a multi-modal cross attention module to obtain a fused embedding;
• performing a sentiment classification on the fused embedding to obtain a sentiment prediction;
• and
• implementing a remediation on the data classification system using the sentiment prediction.

A system, comprising:

• a processor;
• and
• memory including instructions, which when executed by the processor, perform a method comprising:
• obtaining, by a data classification system, raw data, associated with a meeting, wherein the raw data the comprises a textual transcript and an audio file of the meeting;
• applying a generative adversarial network (GAN) to the audio file to obtain audio embeddings and extracted text;
• applying the textual transcripts and the extracted text to a trained language model to obtain textual embeddings;
• applying the audio embeddings and the textual embeddings to a multi-modal cross attention module to obtain a fused embedding;
• performing a sentiment classification on the fused embedding to obtain a sentiment prediction;
• and
• implementing a remediation on the data classification system using the sentiment prediction.