Interactive systems and methods

A method of producing an avatar video, the method comprising the steps of:

• providing a reference image of a person's face;
• providing a plurality of characteristic features representative of a facial model X0 of the person's face, the plurality of characteristic features defining a facial pose dependent on a person speaking;
• providing a target phrase to be rendered over a predetermined time period during the avatar video and providing a plurality of time intervals t within the predetermined time period;
• generating, for each of said plurality of time intervals t, speech features from the target phrase, to provide a sequence of speech features;
• and
• generating, using the plurality of characteristic features and sequence of speech features, a sequence of facial models Xt for each of said plurality of time intervals t, wherein the sequence of facial models Xt is generated using a recursive model.

A system for producing an avatar video, a method comprising the steps of:

• an image processing model for receiving a reference image of a person's face and for extracting a plurality of characteristic features representative of an initial facial model XO of the person's face, wherein the plurality of characteristic features of the initial facial model XO comprise at least one landmark and at least one latent descriptor representing an abstract appearance feature, the plurality of characteristic features defining facial position and facial pose dependent on the person speaking;
• a speech processing module for extracting a target phrase to be rendered over a predetermined time period during the avatar video and for providing a plurality of time intervals t within the predetermined time period;
• the speech processing module configured to generate, for each of said plurality of time intervals t, speech features from the target phrase, to provide a sequence of speech features, the speech features representing abstract quantifiers of audio and linguisic information;
• and
• an avatar rendering module configured to use a recursive model comprising a sequence-to-sequence encoder decoder method to generate, from the initial facial model XO and the sequence of speech features, a sequence of expected facial models for each of said plurality of time intervals t, wherein physical spatio-temporal dynamics of a facial model at each of said plurality of time intervals t are generated by solving a system of ordinary differential equations, ODEs, an expected facial position being derived from a recursive transformation of the speech features and the facial position of the facial model at a current time interval of said plurality of time intervals, which is sampled and being combined with the plurality of characteristic features of the facial model Xt at the current time interval to obtain the plurality of characteristic features of a next facial model Xt+1 in the sequence of expected facial models;
• wherein the avatar rendering module comprises a frame generator configured to combine and decode a sequence of facial models Xt with the initial facial model X0 to generate a sequence of face images to produce the avatar video.

A system for producing an avatar video, a method comprising the steps of:

• an image processing model for receiving a reference image of a person's face and for extracting a plurality of characteristic features representative of a facial model XO of the person's face, the characteristic features defining a facial pose dependent on a person speaking;
• a speech processing module for extracting a target phrase to be rendered over a predetermined time period during the avatar video and for providing a plurality of time intervals t within the predetermined time period;
• the speech processing module configured to generate, for each of said plurality of time intervals t, speech features from the target phrase, to provide a sequence of speech features;
• and
• an avatar rendering module for generating, using the plurality of characteristic features and sequence of speech features, a sequence of facial models Xt for each of said time intervals t, wherein the sequence of facial models Xt is generated using a recursive model.