Detecting emotional state of a user based on facial appearance and visual perception information

A method for detecting an emotional state of a user, the method comprising:

• obtaining, by a processor, a first data stream indicative of facial appearance and gaze direction of the user as the user is viewing a scene;
• determining, by the processor based on the first data stream, a facial expression feature vector indicative of emotional facial expression of the user as the user is viewing the scene;
• obtaining, by the processor, a second data stream indicative of visual content in a field of view of the user as the user is viewing the scene;
• determining, by the processor based on the second data stream, a visual content feature vector indicative of the visual content in the scene;
• fusing, by the processor, the facial expression feature vector with the visual content feature vector to generate a fused feature vector that includes fused features of both the emotional facial expression of the user and the visual content in the scene;
• analyzing, by the processor, the fused feature vector using a neural network trained to provide a scaling vector, the scaling vector including respective scalars that reflect respective degrees of importance of respective channels in the fused feature vector;
• determining, by the processor, emotional state information based on analyzing the fused feature vector scaled based on the scaling vector, wherein the emotional state information i) is indicative of the emotional state of the user and ii) includes a semantic representation of the visual content in the scene, wherein the semantic representation indicates a cause for the emotional state of the user;
• and
• performing, by the processor, an operation with respect to the emotional state information.

A system comprising:

• a first sensor configured to generate a first data stream indicative of facial appearance and gaze direction of a user as the user is viewing a scene;
• a second sensor configured to generate a second data stream indicative of visual content in a field of view of the user as the user is viewing the scene;
• and
• an emotion analysis engine implemented on one or more integrated circuits, the emotion analysis engine configured to:
• obtain the first data stream from the first sensor;
• determine, based on the first data stream, a facial expression feature vector indicative of emotional facial expression of the user as the user is viewing the scene;
• obtain the second data stream from the second sensor;
• determine, based on the second data stream, a visual content feature vector indicative of the visual content in the scene;
• fuse the facial expression feature vector with the visual content feature vector to generate a fused feature vector that includes fused features of both the emotional facial expression of the user and the visual content in the scene;
• analyze the fused feature vector using a neural network trained to provide a scaling vector, the scaling vector including respective scalars that reflect respective degrees of importance of respective channels in the fused feature vector;
• determine emotional state information based on analyzing the fused feature vector scaled based on the scaling vector, wherein the emotional state information is indicative of an emotional state of the user;
• and
• perform an operation with respect to the emotional state information.