System and method of facial analysis

A method of facial analysis comprising:

• recording a video of a subject's face using a camera, the video having a plurality of frames;
• storing the video in a data storage device;
• extracting a pose of the subject's face, the pose including the angle and position of the subject's face;
• extracting a representation of the subject's facial gesture state, the representation including facial keypoints, the facial keypoints being a collection of points on the subject's face;
• concatenating each data stream to align the data streams in time;
• extracting a plurality of facial syllables from the aligned data streams;
• compiling the plurality of facial syllables into a series of state sequences;
• and
• extracting a behavioral fingerprint for the subject based on the series of state sequences, the behavioral fingerprint providing a summary of the subject's state over a given period of time.

A system for subject analysis comprising:

• a camera configured to take a video of a portion of a subject, the video having a plurality of frames;
• a data storage device configured to receive and store the video;
• and
• a data processing system having a processor and program code which when executed:
• (a) extracts a pose of the portion of the subject, the pose including the angle and position of the portion of the subject,
• (b) extracts a representation of the subject's gesture state, the representation including keypoints, the keypoints being a collection of points on the portion of the subject,
• (c) concatenates each data stream to align the data streams in time,
• (d) extracts a plurality of syllables from the aligned data streams,
• (e) compiles the plurality of syllables into a series of state sequences, and
• (f) extracts a behavioral fingerprint for the subject based on the series of state sequences, the behavioral fingerprint providing a summary of the subject's state over a given period of time.