Systems and methods for predicting an emotion based on a multimodal input

An apparatus for predicting an emotion based a multimodal input, the apparatus comprising:

• a communications circuitry configured to receive a multimodal input from a user including at least (i) an amount of keystrokes over a period of time, (ii) text, and (iii) speech;
• an automatic speech recognition circuitry configured to convert the speech to converted text;
• and
• an emotion prediction circuitry configured to:
• cause, by a speech-context Bidirectional Long Short-Term Memory (BLSTM) and using the text and the converted text, generation of context hidden vectors,
• cause generation of audio hidden vectors using the speech and an audio BLSTM,
• generate, by a trained machine learning model using the multimodal input, the context hidden vectors, and the audio hidden vectors, an EmotionPrint for the user, and
• determine a next action based on the EmotionPrint.

A computer program product for predicting a customer's emotions, the computer program product comprising at least one non-transitory computer-readable storage medium storing software instructions that, when executed, cause an apparatus to:

• receive a multimodal input from a user including at least (i) an amount of keystrokes over a period of time, (ii) text, and (iii) speech;
• convert the speech to converted text;
• cause, by a speech-context Bidirectional Long Short-Term Memory (BLSTM) and using the text and the converted text, generation of context hidden vectors;
• cause generation of audio hidden vectors using the speech and an audio BLSTM;
• generate, by a trained machine learning model using the multimodal input, the context hidden vectors, and the audio hidden vectors, an EmotionPrint of the user;
• and
• determine a next action based on the EmotionPrint.