Visual dubbing using synthetic models

A computer-implemented method of processing input video data comprising target footage of a target object of a given class, comprising:

• training an encoder-decoder network comprising an encoder network, a first decoder network, and a second decoder network, the training comprising:
• training a first path through the encoder-decoder network to reconstruct the target footage of the target object, the first path comprising the encoder network and the first decoder network;
• and
• training a second path through the encoder-decoder network to process renders of a synthetic model of an object of the given class exhibiting a range of poses and deformations to determine parameter values for the synthetic model corresponding to the range of poses and deformations, the second path comprising the encoder network and the second decoder network;
• and
• processing, using a trained network path comprising or trained using the encoder network and comprising the first decoder network, source data representing the synthetic model exhibiting a source sequence of deformations, to generate output video data in which the target object exhibits the source sequence of deformations.

One or more non-transient storage media comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out operations comprising:

• obtaining an encoder network trained to process video data depicting objects of a common class to generate respective data arrays representing generic features of the objects of the common class;
• obtaining training video data comprising footage of a target object of the common class;
• training, using the training video data, a target encoder-decoder network comprising the encoder network and a target decoder network to reconstruct the footage of the target object;
• processing, using a target network comprising or trained using the encoder network and comprising the target decoder network, source data representing a synthetic model of an object of the common class exhibiting a source sequence of expressions, to generate output video data in which the target object exhibits the source sequence of deformations.