# US 12,192,593 B2 · Utilizing generative models for resynthesis of transition frames in clipped digital videos > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,593 B2 - Document ID: US12192593B2 - Status: Granted - Kind: B2 - Application number: 18/164,348 - Filed: 2023-02-03 - Published: 2024-08-08 - Granted: 2025-01-07 - Earliest claimed: 2023-02-03 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Xiaojuan Wang; Richard Zhang; Taesung Park; Yang Zhou; Elya Shechtman - Web page: https://patentgrep.com/patent/US12192593B2 ## CPC - H04N 21/8153: {comprising still images, e.g. texture, background image} - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06V 10/82: using neural networks - H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates to systems, methods, and non-transitory computer-readable media that utilize machine learning to generate a sequence of transition frames for a gap in a clipped digital video. For example, the disclosed system receives a clipped digital video that includes a pre-cut frame prior to a gap in the clipped digital video and a post-cut frame following the gap in the clipped digital video. Moreover, the disclosed system utilizes a natural motion sequence model to generates a sequence of transition keypoint maps between the pre-cut frame and the post-cut frame. Furthermore, using a generative neural network, the disclosed system generates a sequence of transition frames for the gap in the clipped digital video from the sequence of transition keypoint maps. ## Claims ### Claim 1 A method comprising: receiving a clipped digital video comprising a pre-cut frame prior to a gap in the clipped digital video and a post-cut frame following the gap in the clipped digital video; generating a sequence of transition keypoint maps utilizing the pre-cut frame and the post-cut frame; and generating, utilizing a generative neural network, a sequence of transition frames for the gap in the clipped digital video from the sequence of transition keypoint maps. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating a pre-cut keypoint map from the pre-cut frame and a post-cut keypoint map from the post-cut frame. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the sequence of transition frames further comprises: generating a source keypoint map from a source frame of the clipped digital video; and generating the sequence of transition frames from the sequence of transition keypoint maps, the source frame of the clipped digital video, and the source keypoint map. ### Claim 4 (depends on 2) The method of claim 2, wherein generating the sequence of transition keypoint maps further comprises: determining, utilizing a natural motion sequence model, the sequence of transition keypoint maps from the pre-cut keypoint map and the post-cut keypoint map. ### Claim 5 (depends on 4) The method of claim 4, wherein determining, utilizing the natural motion sequence model, the sequence of transition keypoint maps comprises determining a natural motion sequence utilizing a transformer neural network. ### Claim 6 (depends on 3) The method of claim 3, wherein generating the sequence of transition frames further comprises, generating a transition frame by: generating, utilizing a neural network encoder, a source feature map from the source frame; and generating a target feature map for the transition frame by combining the source feature map, the source keypoint map, and a keypoint map corresponding to the transition frame. ### Claim 7 (depends on 6) The method of claim 6, further comprising generating the transition frame utilizing the generative neural network from the target feature map and the source frame. ### Claim 8 (depends on 6) The method of claim 6, further comprising: generating, utilizing the neural network encoder, a plurality of source feature maps from a plurality of source frames; and generating the target feature map for the transition frame from the plurality of source feature maps from the plurality of source frames, a plurality of source keypoint maps, and the keypoint map corresponding to the transition frame. ### Claim 9 A system comprising: one or more memory components; and one or more processing devices coupled to the one or more memory components, the one or more processing devices to perform operations comprising: receiving a clipped digital video comprising a pre-cut frame prior to a gap in the clipped digital video and a post-cut frame following the gap in the clipped digital video; generating, a pre-cut keypoint map from the pre-cut frame and a post-cut keypoint map from the post-cut frame; generating a sequence of transition keypoint maps for the gap in the clipped digital video from the pre-cut keypoint map and the post-cut keypoint map; and generating, utilizing a generative neural network, a sequence of transition frames for the clipped digital video from the sequence of transition keypoint maps, a source frame of the clipped digital video, and a keypoint map corresponding to the source frame. ### Claim 10 (depends on 9) The system of claim 9, wherein the operations further comprise generating the sequence of transition keypoint maps from the pre-cut keypoint map and the post-cut keypoint map by utilizing a transformer neural network. ### Claim 11 (depends on 9) The system of claim 9, wherein generating the sequence of transition frames further comprises generating, utilizing the generative neural network, the sequence of transition frames from the sequence of transition keypoint maps, a source frame of the clipped digital video, and a source keypoint map generated from the source frame. ### Claim 12 (depends on 9) The system of claim 9, wherein generating the sequence of transition frames further comprises, generating a transition frame by: generating, utilizing a neural network encoder, a source feature map from the source frame; generating a target feature map for the transition frame by utilizing a feature splatting model to combine the source feature map, a source keypoint map corresponding to the source frame, and a keypoint map corresponding to the transition frame; and generating, utilizing the generative neural network, the transition frame from the target feature map and the source frame. ### Claim 13 (depends on 9) The system of claim 9, wherein the operations further comprise learning parameters of the generative neural network based on a reconstruction loss and a temporal discriminator loss. ### Claim 14 (depends on 10) The system of claim 10, wherein the operations further comprise learning parameters of the transformer neural network by: determining a measure of loss by comparing the sequence of transition keypoint maps for the gap with a ground truth sequence of transition keypoint maps extracted frames of the gap in the clipped digital video; and modifying the parameters of the transformer neural network based on the measure of loss. ### Claim 15 (depends on 12) The system of claim 12, wherein generating the transition frame further comprises: generating, utilizing the neural network encoder, a plurality of source feature maps from a plurality of source frames; and combining the plurality of source feature maps to generate the transition frame. ### Claim 16 (depends on 15) The system of claim 15, wherein generating the transition frame further comprises generating the target feature map for the transition frame from the plurality of source feature maps corresponding to the plurality of source frames, a plurality of source keypoint maps corresponding to the plurality of source frames, and the keypoint map corresponding to the transition frame. ### Claim 17 A non-transitory computer-readable medium storing executable instructions which, when executed by at least one processing device, cause the at least one processing device to perform operations comprising: receiving a clipped digital video comprising a pre-cut frame prior to a gap in the clipped digital video and a post-cut frame following the gap in the clipped digital video; generating a sequence of transition keypoint maps utilizing the pre-cut frame and the post-cut frame; and generating, utilizing a generative neural network, a sequence of transition frames for the gap in the clipped digital video from the sequence of transition keypoint maps. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein generating the sequence of transition keypoint maps further comprises: determining the sequence of transition keypoint maps from a pre-cut keypoint map corresponding to the pre-cut frame and a post-cut keypoint map corresponding to the post-cut frame by utilizing a natural motion sequence model. ### Claim 19 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein generating the sequence of transition frames further comprises: generating a plurality of source feature maps from a plurality of source frames; generating target feature maps for the sequence of transition frames from the plurality of source feature maps of the plurality of source frames, a plurality of source keypoint maps of the plurality of source frames, and the sequence of transition keypoint maps corresponding to the sequence of transition frames; and generating the sequence of transition frames utilizing the generative neural network from the target feature maps. ### Claim 20 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the operations further comprise training the generative neural network by: generating, utilizing a temporal discriminator neural network, a transition sequence authenticity prediction from the sequence of transition frames; determining a transition sequence measure of loss based on the transition sequence authenticity prediction; and modifying parameters of the generative neural network based on the transition sequence measure of loss.