Actor-replacement system for videos

A method comprising:

• estimating, by a computing system using a skeletal detection model, a pose of an original actor for each of multiple frames of a video;
• obtaining, by the computing system for each of a plurality of the estimated poses of the original actor, a respective image of a replacement actor;
• obtaining, by the computing system, replacement speech in the replacement actor's voice that corresponds to speech of the original actor in the video;
• generating, by the computing system using the estimated poses, the images of the replacement actor, and the replacement speech, synthetic frames corresponding to the multiple frames of the video that depict the replacement actor in place of the original actor, wherein the synthetic frames depict the replacement actor in respective poses that align with the estimated poses of the original actor in corresponding frames of the video, and wherein the synthetic frames comprise facial expressions for the replacement actor that temporally align with the replacement speech, wherein generating the synthetic frames comprises, for a given frame of the multiple frames, inserting, using an object insertion model, an image of the replacement actor into the given frame at a location indicated by the estimated pose of the original actor so as to obtain a modified frame, and wherein generating the synthetic frames further comprises providing the replacement speech and the modified frame as input to a temporal generative adversarial network having an ensemble of discriminators;
• and
• combining the synthetic frames and the replacement speech so as to obtain a synthetic video that replaces the original actor with the replacement actor.

A non-transitory computer-readable medium having stored thereon program instructions that upon execution by a computing system, cause performance of a set of acts comprising:

• estimating, using a skeletal detection model, a pose of an original actor for each of multiple frames of a video;
• obtaining, for each of a plurality of the estimated poses of the original actor, a respective image of a replacement actor;
• obtaining replacement speech in the replacement actor's voice that corresponds to speech of the original actor in the video;
• generating, using the estimated poses, the images of the replacement actor, and the replacement speech, synthetic frames corresponding to the multiple frames of the video that depict the replacement actor in place of the original actor, wherein the synthetic frames depict the replacement actor in respective poses that align with the estimated poses of the original actor in corresponding frames of the video, and wherein the synthetic frames comprise facial expressions for the replacement actor that temporally align with the replacement speech, wherein generating the synthetic frames comprises, for a given frame of the multiple frames, inserting, using an object insertion model, an image of the replacement actor into the given frame at a location indicated by the estimated pose of the original actor so as to obtain a modified frame, and wherein generating the synthetic frames further comprises providing the replacement speech and the modified frame as input to a temporal generative adversarial network having an ensemble of discriminators;
• and
• combining the synthetic frames and the replacement speech so as to obtain a synthetic video that replaces the original actor with the replacement actor.