3D facial capture and modification using image and temporal tracking neural networks

A system comprising:

• a memory to store a plurality of sequential video images;
• and
• a processor coupled to the memory, the processor to:
• apply a first convolutional neural network (CNN) to a first input vector corresponding to a first video image of the plurality of sequential video images to generate a first morphable model parameter vector;
• apply a second CNN to a second input vector having chroma or luma or both image data of both the first video image and a second video image in sequence with the first video image in the plurality of sequential video images to generate a morphable model parameter delta vector;
• generate a first face model of the human face by applying the first morphable model parameter vector to a 3D face model;
• determine a second morphable model parameter vector by modifying the first morphable model parameter vector using the morphable model parameter delta vector;
• generate a second face model of the human face by applying the second morphable model parameter vector to the 3D face model;
• and
• transmit output video images comprising image features based at least in part on the first and second face models for storage or presentment.

At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a device, cause the device to process video by:

• receiving a plurality of sequential video images comprising representations of a human face;
• applying a first convolutional neural network (CNN) to a first input vector corresponding to a first video image of the plurality of sequential video images to generate a first morphable model parameter vector;
• applying a second CNN to a second input vector having chroma or luma or both image data of both the first video image and a second video image in sequence with the first video image in the plurality of sequential video images to generate a morphable model parameter delta vector;
• generating a first face model of the human face by applying the first morphable model parameter vector to a 3D face model;
• determining a second morphable model parameter vector by modifying the first morphable model parameter vector using the morphable model parameter delta vector;
• generating a second face model of the human face by applying the second morphable model parameter vector to the 3D face model;
• and
• storing or presenting output video images comprising image features based at least in part on the first and second face models.