Facial features tracker with advanced training for natural rendering of human faces in real-time

A method, comprising:

• detecting an image of a face in a frame of a video stream;
• analyzing the image of the face for yaw and pitch angles associated with a head orientation of the face;
• selecting a head orientation class from a comprehensive set of head orientation classes based on the yaw and pitch angles, each head orientation class providing a set of feature points comprising a deformable 3D model of averaged facial features at head orientation angles characteristic of the head orientation class;
• modifying the deformable 3D model of the averaged facial features from the selected head orientation class into a modified 3D model of the image of the face by calculating real head orientation angles and estimate rotational, scaling, and translational factors to apply to the deformable 3D model of averaged facial features;
• calculating a 2D model of tracking points representing the image of the face, the tracking points projected from the modified 3D model onto an image plane;
• and
• controlling, actuating, or animating a piece of hardware based on the 2D model of projected tracking points.