Systems and methods for learning emotion representations from verbal and nonverbal communication
AssigneeThe Penn State Research Foundation
InventorsJames Z. Wang, Chenyan Wu, Sitao Zhang, Yimu Pan
Life of this patent
Estimating the term…
Abstract
Systems and methods for understanding emotions, including static and dynamically changing emotional states, from verbal and nonverbal communication, using only uncurated data are provided herein. Systems and methods for bodily expressed emotion understanding (BEEU), incorporating motor element analysis, are also provided herein. Systems and methods for context-aware emotion recognition in an uncontrolled real-world setting allowing for simultaneous processing of multiple subjects of interest within a single scene are also provided herein.
Claims20
A method of understanding emotions, including static and dynamically changing emotional states, and from stored or live visual information, the method comprising the steps of:
- obtaining a paired nonverbal and verbal information dataset consisting of nonverbal input and verbal input associated with the nonverbal input using uncurated data from daily communication;
- using a subject-aware context encoding strategy to process the nonverbal input taking into account interaction between a subject of interest and a corresponding context;
- using a text encoder to process the verbal input;
- and
- training a machine learning model to learn a consistency between the nonverbal input and the associated verbal input by minimizing a sentiment-guided contrastive loss £ for understanding emotions.
A method of bodily expressed emotion understanding (BEEU), for both static and dynamically changing emotional states, and from stored or live visual information, using motor elements, the method comprising the steps of:
- providing a BEEU dataset consisting of video clips each having emotion category labels;
- providing a body motor elements dataset (BoME) consisting of a second set of video clips each annotated with motor element labels, the body motor elements having correspondence with emotion categories;
- training a deep neural network on the BEEU dataset and the BoME dataset to recognize the body motor elements and use the body motor elements as intermediate features to recognize the emotion categories;
- providing a dual-branch network structure including an emotion branch and a motor element branch, allowing the dual-branch network structure to output the motor element labels and the emotion category labels simultaneously;
- deploying a fusion operation by adding the motor element labels as input to emotion recognition of the emotion branch, whereby the emotion category labels are predicted with the assistance of motor element branch;
- and
- utilizing a bridge loss to allow the motor element labels to supervise emotion prediction, based on the correspondence between the body motor elements and the emotion categories.
A method of context-aware emotion recognition, for both static and dynamically changing emotional states, and from stored or live visual information, in an uncontrolled real-world setting allowing for simultaneous processing of multiple subjects of interest within a single image, comprising the steps of:
- processing the entire image, or an image frame from a video, to extract spatial features in a single operation, regardless of the number of subjects of interest present within the image;
- embedding positional information of each subject of interest in the image;
- retrieving relevant spatial features in a parallel manner for each subject and context query from the image, by a cross-attention operation, the embedded positional prompts serving to direct a decoder to correspond to the specific subject of interest and discriminate it from others present in the image, thereby using a single self-attention operation to capture both inter-subject and subject-context dynamics.
A system for context-aware emotion recognition in an uncontrolled real-world setting allowing for simultaneous processing of multiple subjects of interest within a single image, comprising:
- transformers representing a class of neural networks that rely on attention mechanisms, including:
- a unified encoder, including:
- an image encoder for extracting spatial features of the entire image in a single operation, regardless of the number of subjects of interest present within the image;
- a prompt encoder embedding the positional information of each subject of interest in the image;
- and
- a context-aware decoder configured to simultaneously decode emotions of all subjects in a parallel manner using the spatial features and positional prompts.
Dependencies are inferred from the claim text.