# US 2021/0243383 A1 · Video synthesis method, model training method, device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243383 A1
- Document ID: US20210243383A1
- Status: Published application
- Kind: A1
- Application number: 17/239,438
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2019-03-06
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Haozhi HUANG; Kun CHENG; Chun YUAN; Wei LIU
- Web page: https://patentgrep.com/patent/US20210243383A1

## CPC

- H04N 5/265: Mixing
- G06F 18/25: Fusion techniques
- G06F 18/253: {of extracted features}
- G06V 10/449: {Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters}
- G06V 10/806: {of extracted features}
- G06V 20/47: {Detecting features for summarising video content}
- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Embodiments of this application disclose methods, systems, and devices for video synthesis. In one aspect, a method comprises obtaining a plurality of frames corresponding to source image information of a first to-be-synthesized video, each frame of the source image information. The method also comprises obtaining a plurality of frames corresponding to target image information of a second to-be-synthesized video. For each frame of the plurality of frames corresponding to the target image information of the second to-be-synthesized video, the method comprises fusing a respective source image from the first to-be-synthesized video, a corresponding source motion key point, and a respective target motion key point corresponding to the frame using a pre-trained video synthesis model, and generating a respective output image in accordance with the fusing. The method further comprises repeating the fusing and the generating steps for the second to-be-synthesized video to produce a synthesized video.

## Claims

### Claim 1

A video synthesis method, comprising:
obtaining a plurality of frames corresponding to source image information of a first to-be-synthesized video, each frame of the source image information comprising a source image that includes a transfer object and a source motion key point having a one-to-one correspondence with the source image;
obtaining a plurality of frames corresponding to target image information of a second to-be-synthesized video, the second to-be-synthesized video having a motion reference sequence for indicating an image order of video synthesis, each frame of the target image information comprising a target motion key point;
for each frame of the plurality of frames corresponding to the target image information of the second to-be-synthesized video:
fusing a respective source image from the first to-be-synthesized video, the corresponding source motion key point, and a respective target motion key point corresponding to the frame using a pre-trained video synthesis model; and
generating a respective output image in accordance with the fusing, the respective output image comprising the transfer object modified by the respective target motion key point and corresponding to a respective frame of the plurality of frames of the source image information and the respective target motion key point; and
repeating the fusing and the generating steps for the second to-be-synthesized video to produce a synthesized video according to the motion reference sequence.

### Claim 2 (depends on 1)

The method according to claim 1, wherein generating the respective output image comprises:
performing feature extraction on the plurality of frames corresponding to the source image information and the first target motion key point using the video synthesis model to obtain a foreground feature set and a background feature set, the foreground feature set comprising a plurality of foreground features, the background feature set comprising a plurality of background features;
obtaining a foreground fusion feature corresponding to the foreground feature set using the video synthesis model;
obtaining a background fusion feature corresponding to the background feature set using the video synthesis model;
performing convolution processing on the foreground fusion feature using the video synthesis model to obtain a foreground image and a foreground mask, the foreground mask being used for extracting an image contour;
performing convolution processing on the background fusion feature using the video synthesis model to obtain a background image; and
processing the foreground image, the foreground mask, and the background image using the video synthesis model to generate the respective output image.

### Claim 3 (depends on 2)

The method according to claim 2, wherein obtaining the foreground fusion feature further comprises:
determining a plurality of frames corresponding to foreground attention mask maps according to the foreground feature set, the source motion key point of each frame of the source image information in the plurality frames corresponding to the source image information, and the respective target motion key point; and
determining the foreground fusion feature according to the plurality of frames of foreground attention mask maps and the foreground feature set.

### Claim 4 (depends on 3)

The method according to claim 3, wherein determining the foreground fusion feature further comprises:
calculating the foreground fusion feature according to:
{tilde over (F)}fgt=Σk=1KFfgtk⊙Afgtk,
wherein {tilde over (F)}fgt represents the foreground fusion feature, {tilde over (F)}fgt represents a kth frame of foreground feature in the foreground feature set, Afgtk represents a kth frame of foreground attention mask map in the plurality of frames of foreground attention mask maps, and ⊙ represents pixel multiplication.

### Claim 5 (depends on 2)

The method according to claim 2, wherein obtaining the background fusion feature further comprises:
determining a plurality of frames of background attention mask maps according to the background feature set, the source motion key point of each frame of the source image information in the plurality of frames corresponding to source image information, and the first target motion key point; and
determining the background fusion feature according to the plurality of frames corresponding to the background attention mask maps and the background feature set.

### Claim 6 (depends on 5)

The method according to claim 5, wherein determining the background fusion feature according to the plurality of frames corresponding to the background attention mask maps and the background feature set comprises:
calculating the background fusion feature according to:
{tilde over (F)}bgt=Σk=1KFbgtk⊙Abgtk,
wherein {tilde over (F)}bgt represents the background fusion feature, Fbgtk represents a kth frame of background feature in the background feature set, Abgtk represents a kth frame of background attention mask map in the K frames of background attention mask maps, and ⊙ represents pixel multiplication.

### Claim 7 (depends on 2)

The method according to claim 2, wherein processing the foreground image, the foreground mask, and the background image using the video synthesis model further comprises:
calculating the first output image according to:
Ot=Ofgt⊙Mfgt+Obgt(1−Mfgt),
Ot represents the first output image corresponding to a tth frame, Ofgt represents the foreground image corresponding to the tth frame, Obgt represents the background image corresponding to the tth frame, Mfgt represents the foreground mask corresponding to the tth frame, and ⊙ represents pixel multiplication.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising prior to producing the synthesized video:
generating a second output image corresponding to the plurality of frames corresponding to the source image information and a second target motion key point using the video synthesis model, the second output image comprising the transfer object, and the second target motion key point being a target motion key point in the plurality of frames corresponding to the target image information; and
producing the synthesized video further comprises:
processing the first output image and the second output image according to the motion reference sequence to obtain the synthesized video.

### Claim 9

A computing device, comprising:
one or more processors; and
memory, the memory storing one or more programs comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining a plurality of frames corresponding to source image information of a first to-be-synthesized video, each frame of the source image information comprising a source image that includes a transfer object and a source motion key point having a one-to-one correspondence with the source image;
obtaining a plurality of frames corresponding to target image information of a second to-be-synthesized video, the second to-be-synthesized video having a motion reference sequence for indicating an image order of video synthesis, each frame of the target image information comprising a target motion key point;
for each frame of the plurality of frames corresponding to the target image information of the second to-be-synthesized video:
fusing a respective source image from the first to-be-synthesized video, the corresponding source motion key point, and a respective target motion key point corresponding to the frame using a pre-trained video synthesis model; and
generating a respective output image in accordance with the fusing, the respective output image comprising the transfer object modified by the respective target motion key point and corresponding to a respective frame of the plurality of frames of the source image information and the respective target motion key point; and
repeating the fusing and the generating steps for the second to-be-synthesized video to produce a synthesized video according to the motion reference sequence.

### Claim 10 (depends on 9)

The computing device according to claim 9, wherein generating the respective output image comprises:
performing feature extraction on the plurality of frames corresponding to the source image information and the first target motion key point using the video synthesis model to obtain a foreground feature set and a background feature set, the foreground feature set comprising a plurality of foreground features, the background feature set comprising a plurality of background features;
obtaining a foreground fusion feature corresponding to the foreground feature set using the video synthesis model;
obtaining a background fusion feature corresponding to the background feature set using the video synthesis model;
performing convolution processing on the foreground fusion feature using the video synthesis model to obtain a foreground image and a foreground mask, the foreground mask being used for extracting an image contour;
performing convolution processing on the background fusion feature using the video synthesis model to obtain a background image; and
processing the foreground image, the foreground mask, and the background image using the video synthesis model to generate the respective output image.

### Claim 11 (depends on 10)

The computing device according to claim 10, wherein obtaining the foreground fusion feature further comprises:
determining a plurality of frames corresponding to foreground attention mask maps according to the foreground feature set, the source motion key point of each frame of the source image information in the plurality frames corresponding to the source image information, and the respective target motion key point; and
determining the foreground fusion feature according to the plurality of frames of foreground attention mask maps and the foreground feature set.

### Claim 12 (depends on 11)

The computing device according to claim 11, wherein determining the foreground fusion feature further comprises:
calculating the foreground fusion feature according to:
{tilde over (F)}fgt=Σk=1KFfgtk⊙Afgtk,
wherein {tilde over (F)}fgt represents the foreground fusion feature, {tilde over (F)}fgt represents a kth frame of foreground feature in the foreground feature set, Afgtk represents a kth frame of foreground attention mask map in the plurality of frames of foreground attention mask maps, and ⊙ represents pixel multiplication.

### Claim 13 (depends on 10)

The computing device according to claim 10, wherein obtaining the background fusion feature further comprises:
determining a plurality of frames of background attention mask maps according to the background feature set, the source motion key point of each frame of the source image information in the plurality of frames corresponding to source image information, and the first target motion key point; and
determining the background fusion feature according to the plurality of frames corresponding to the background attention mask maps and the background feature set.

### Claim 14 (depends on 13)

The computing device according to claim 13, wherein determining the background fusion feature according to the plurality of frames corresponding to the background attention mask maps and the background feature set comprises:
calculating the background fusion feature according to:
{tilde over (F)}bgt=Σk=1KFbgtk⊙Abgtk,
wherein {tilde over (F)}bgt represents the background fusion feature, Fbgtk represents a kth frame of background feature in the background feature set, Abgtk represents a kth frame of background attention mask map in the K frames of background attention mask maps, and ⊙ represents pixel multiplication

### Claim 15

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform operations comprising:
obtaining a plurality of frames corresponding to source image information of a first to-be-synthesized video, each frame of the source image information comprising a source image that includes a transfer object and a source motion key point having a one-to-one correspondence with the source image;
obtaining a plurality of frames corresponding to target image information of a second to-be-synthesized video, the second to-be-synthesized video having a motion reference sequence for indicating an image order of video synthesis, each frame of the target image information comprising a target motion key point;
for each frame of the plurality of frames corresponding to the target image information of the second to-be-synthesized video:
fusing a respective source image from the first to-be-synthesized video, the corresponding source motion key point, and a respective target motion key point corresponding to the frame using a pre-trained video synthesis model; and
generating a respective output image in accordance with the fusing, the respective output image comprising the transfer object modified by the respective target motion key point and corresponding to a respective frame of the plurality of frames of the source image information and the respective target motion key point; and
repeating the fusing and the generating steps for the second to-be-synthesized video to produce a synthesized video according to the motion reference sequence.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein generating the respective output image comprises:
performing feature extraction on the plurality of frames corresponding to the source image information and the first target motion key point using the video synthesis model to obtain a foreground feature set and a background feature set, the foreground feature set comprising a plurality of foreground features, the background feature set comprising a plurality of background features;
obtaining a foreground fusion feature corresponding to the foreground feature set using the video synthesis model;
obtaining a background fusion feature corresponding to the background feature set using the video synthesis model;
performing convolution processing on the foreground fusion feature using the video synthesis model to obtain a foreground image and a foreground mask, the foreground mask being used for extracting an image contour;
performing convolution processing on the background fusion feature using the video synthesis model to obtain a background image; and
processing the foreground image, the foreground mask, and the background image using the video synthesis model to generate the respective output image.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium according to claim 16, wherein obtaining the foreground fusion feature further comprises:
determining a plurality of frames corresponding to foreground attention mask maps according to the foreground feature set, the source motion key point of each frame of the source image information in the plurality frames corresponding to the source image information, and the respective target motion key point; and
determining the foreground fusion feature according to the plurality of frames of foreground attention mask maps and the foreground feature set.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium according to claim 17, wherein determining the foreground fusion feature further comprises:
calculating the foreground fusion feature according to:
{tilde over (F)}fgt=Σk=1KFfgtk⊙Afgtk,
wherein {tilde over (F)}fgt represents the foreground fusion feature, {tilde over (F)}fgt represents a kth frame of foreground feature in the foreground feature set, Afgtk represents a kth frame of foreground attention mask map in the plurality of frames of foreground attention mask maps, and ⊙ represents pixel multiplication.

### Claim 19 (depends on 16)

The non-transitory computer-readable storage medium according to claim 16, wherein obtaining the background fusion feature further comprises:
determining a plurality of frames of background attention mask maps according to the background feature set, the source motion key point of each frame of the source image information in the plurality of frames corresponding to source image information, and the first target motion key point; and
determining the background fusion feature according to the plurality of frames corresponding to the background attention mask maps and the background feature set.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium according to claim 19, wherein determining the background fusion feature according to the plurality of frames corresponding to the background attention mask maps and the background feature set comprises:
calculating the background fusion feature according to:
{tilde over (F)}bgt=Σk=1KFbgtk⊙Abgtk,
wherein {tilde over (F)}bgt represents the background fusion feature, Fbgtk represents a kth frame of background feature in the background feature set, Abgtk represents a kth frame of background attention mask map in the K frames of background attention mask maps, and ⊙ represents pixel multiplication.