Emotion and action recognition ai
OwnerMuhammad Waqas, Muhammad Usmaninventors
Life of this patent
Estimating the term…
Abstract
In one aspect, a method for emotion and action recognition, comprising: receiving, at one or more processors, image data from a video stream; analyzing, using a convolutional neural network, the image data to identify facial expressions for emotion recognition; extracting, using a landmark detection model, a plurality of body landmarks from the image data; determining, based on the plurality of body landmarks, spatial relationships between body joints over a sequence of frames; classifying, using a long short-term memory (LSTM) network, an action based on changes in the spatial relationships across the sequence of frames; and outputting both an emotional state classification and an action classification based on the analyzing and the classifying.
Claims19
A method for emotion and action recognition, comprising:
- receiving, at one or more processors, image data from a video stream;
- analyzing, using a convolutional neural network, the image data to identify facial expressions for emotion recognition;
- extracting, using a landmark detection model, a plurality of body landmarks from the image data;
- determining, based on the plurality of body landmarks, spatial relationships between body joints over a sequence of frames;
- classifying, using a long short-term memory (LSTM) network, an action based on changes in the spatial relationships across the sequence of frames;
- and outputting both an emotional state classification and an action classification based on the analyzing and the classifying.
A system for emotion and action recognition, comprising:
- one or more processors;
- a memory coupled to the one or more processors;
- an image capture device configured to provide image data;
- a convolutional neural network implemented by the one or more processors and configured to analyze facial expressions in the image data;
- a landmark detection model implemented by the one or more processors and configured to extract body landmarks from the image data;
- a long short-term memory (LSTM) network implemented by the one or more processors and configured to classify actions based on the body landmarks;
- and wherein the one or more processors are configured to output both emotional state classifications and action classifications based on the analyzed facial expressions and classified actions.
An emotion and action recognition system comprising:
- a computer vision module configured to receive image data from video streams;
- a neural network configured to identify facial expressions from the received image data;
- a landmark detection model configured to extract body landmarks for action classification from the received image data;
- a memory storing instructions;
- and a processor configured to execute the instructions to:
- capture video frames from at least one of a camera or a video file, convert each frame into a binary input array, apply deep learning techniques to estimate poses and classify emotions into at least five emotional states, process a sequence of frames using a recurrent neural network to output classified actions based on changes in spatial relationships, store results of pose estimations as global variables in the memory, access the stored results using application programming interfaces for further processing, and implement optimization techniques including at least batch normalization and data augmentation during preprocessing to improve accuracy and reduce latency.
Dependencies are inferred from the claim text.