Real-time tracking of facial features in unconstrained video

A computer implemented method for generating a set of feature locations characterizing a face of an animal subject from an unconstrained video sequence that constitutes a physical record of the face of the subject, the method implemented using computer processes comprising:

• in a face-detecting process, delineating, by way of region-bounding coordinates, the face of the subject within an image selected from the sequence;
• detecting, in the selected image, a small set of landmarks, the set having fewer than 10 landmarks, corresponding to the face of the subject, using a convolutional neural network, trained to take as input an image region corresponding to the face of the subject and to return a set of coordinates at computational speeds approximating real time;
• projectively fitting a three-dimensional character model to the detected landmarks, and using the fitted model to estimate physical locations of additional landmarks, so as to provide a complete hypothesized set of facial landmarks;
• and
• in a calibration process and using a set of tracked feature locations, refining the three-dimensional character model so that it more closely matches a shape of a head of the subject;
• and
• in a feature tracker process, updating the hypothesized set of facial landmarks to improve convergence between predicted feature locations and their actual physical locations based on data sampled from the selected image.