Video reenactment with hair shape and motion transfer

A method for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said method comprising steps of:

• encoding face and hair identity information of the source image to produce an identity vector, where said face and hair identity information does not vary over time;
• encoding pose and illumination information of the destination video to produce a pose and illumination vector;
• encoding hair motion information of the destination video to produce a hair motion vector, where said hair motion information varies over time;
• and
• feeding the identify vector, pose and illumination vector, and hair motion vector as three inputs to a neural network generator, said neural network generator generating in response to the three inputs a composite video comprising face and hair information from the source image inserted into the destination video.

An apparatus for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said apparatus comprising:

• an identity encoder configured to encode face and hair information of the source image and to produce as an output an identity vector, where said face and hair information does not comprise any motion information;
• a pose encoder configured to encode pose information of the destination video and to produce as an output a pose vector;
• an illumination encoder configured to encode head and hair illumination of the destination video, and to produce as an output an illumination vector;
• and
• a hair motion encoder configured to encode hair motion information of the destination video, and to produce as an output a hair motion vector, where said hair motion information varies over time;
• wherein
• the identity vector, pose vector, illumination vector, and hair motion vector are fed as four inputs to a neural network generator plus sequencer, wherein the neural network generator plus sequencer, in response to the four inputs, produces a composite video comprising face and hair information of the source image inserted into the destination video.

A method for inserting face and hair information from a source image into a destination video while mimicking pose, illumination, and hair motion of the destination video, said method comprising steps of:

• encoding face and hair identity information of the source image to produce an identity vector, where said face and hair identity information does not vary over time;
• activating a pose and illumination encoder to encode pose and illumination information of the destination video;
• activating a hair motion encoder to encode hair motion information of the destination video, where said hair motion information varies over time;
• and
• feeding the identity vector, an output of the pose and illumination vector, and an output of the hair motion encoder as three inputs to a neural network generator plus sequencer, wherein the neural network generator plus sequencer, in response to the three inputs, produces a composite video comprising face and hair information of the source image inserted into the destination video.