# US 2022/0284642 A1 · Method for training convolutional neural network, and method and device for stylizing video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0284642 A1 - Document ID: US20220284642A1 - Status: Published application - Kind: A1 - Application number: 17/825,312 - Filed: 2022-05-26 - Published: 2022-09-08 - Earliest claimed: 2019-11-27 - Assignees: Guangdong Oppo Mobile Telecommunications Corp., Ltd. - Applicants: Guangdong Oppo Mobile Telecommunications Corp., Ltd. - Inventors: Jenhao HSIAO - Web page: https://patentgrep.com/patent/US20220284642A1 ## CPC - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence - G06N 3/048: Activation functions - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided are a method and device for stylizing video as well as a method for training a convolutional neural network (CNN). In the method, each of a plurality of original frames of the video is transformed into a stylized frame by using a first CNN for stylizing; at least one first loss is determined according to a first original frame and second original frame of the plurality of original frames, the second original frame being next to the first original frame; the first CNN is trained according to at least one first loss; and the video is stylized by using the trained first CNN. ## Claims ### Claim 1 A method for training a convolutional neural network (CNN) for stylizing a video, comprising: transforming each of a plurality of original frames of the video into a stylized frame by using a first CNN for stylizing; determining at least one first loss according to a first original frame and second original frame of the plurality of original frames and results of the transforming, the second original frame being next to the first original frame; and training the first CNN according to the at least one first loss. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one first loss comprises a semantic-level temporal loss, and the determining at least one first loss comprises: extracting a first output of a hidden layer in the first CNN when the first CNN is applied to the first original frame, and extracting a second output of the hidden layer in the first CNN when the first CNN is applied to the second original frame; and determining a semantic-level temporal loss according to a first difference between the first output and the second output. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one first loss comprises a contrastive loss, and the determining at least one first loss comprises: determining a contrastive loss according to a second difference between: (a) a difference between the first original frame and a stylized first frame corresponding to the first original frame, and (b) a difference between the second original frame and a stylized second frame corresponding to the second original frame. ### Claim 4 (depends on 1) The method of claim 1, further comprising: transforming each of the plurality of original frames of the video by using a second CNN, the second CNN having been trained on an ImageNet dataset; transforming each of a plurality of the stylized frames by using the second CNN; determining at least one second loss according to an output feature vector of each of the plurality of the original frames at a first layer of the second CNN, and an output feature vector of each of the plurality of the stylized frames at a first layer of the second CNN, wherein training the first CNN according to the at least one first loss comprises training the first CNN according to the at least one first loss and the at least one second loss. ### Claim 5 (depends on 4) The method of claim 4, wherein the at least one second loss comprises a content loss, and the method further comprises: extracting a first feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to each of the plurality of original frames; extracting a second feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to a stylized frame corresponding to the original frame; and determining the content loss according to Euclidean distance between the first feature map and second feature map. ### Claim 6 (depends on 4) The method of claim 4, wherein the at least one second loss comprises a style loss, and the method further comprises: determining a first Gram matrix according to a first feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to each of the plurality of original frames; determining a second Gram matrix according to a second feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to a stylized frame corresponding to the original frame; and determining the style loss according a difference between the first Gram matrix and second Gram matrix. ### Claim 7 (depends on 6) The method of claim 6, wherein determining the style loss according the difference between the first Gram matrix and second Gram matrix comprises: determining the style loss according a squared Frobenius norm of the difference between the first Gram matrix and second Gram matrix. ### Claim 8 (depends on 6) The method of claim 6, wherein training the first CNN according to the at least one first loss and the at least one second loss comprises: training the first CNN such that a weighted sum of the at least one first loss and the at least one second loss is minimized. ### Claim 9 (depends on 8) The method of claim 8, wherein training the first CNN such that a weighted sum of the at least one first loss and the at least one second loss is minimized comprises: training the first CNN based on a method which uses gradient to update network parameters of the first CNN, such that a weighted sum of the at least one first loss and the at least one second loss is minimized. ### Claim 10 (depends on 4) The method of claim 4, wherein the second CNN is selected from a group comprising a VGG network, InceptionNet, and ResNet. ### Claim 11 A method for stylizing a video, comprising: stylizing a video by using a first convolutional neural network (CNN); wherein the first CNN has been trained according to at least one first loss which is determined according to a first original frame and second original frame of a plurality of original frames of the video and results of transforming, the second original frame being next to the first original frame, the transforming comprising: transforming each of a plurality of original frames of the video into a stylized frame by using a first CNN for stylizing. ### Claim 12 (depends on 11) The method of claim 11, wherein the at least one first loss comprises a semantic-level temporal loss, and the semantic-level temporal loss is determined according to a first difference between a first output and a second output, the first output is an output of a hidden layer in the first CNN when the first CNN is applied to the first original frame, and the second output is an output of the hidden layer in the first CNN when the first CNN is applied to the second original frame. ### Claim 13 (depends on 11) The method of claim 11, wherein the at least one first loss comprises a contrastive loss, and the contrastive loss is determined according to a second difference between: (a) a difference between the first original frame and a stylized first frame corresponding to the first original frame, and (b) a difference between the second original frame and a stylized second frame corresponding to the second original frame. ### Claim 14 (depends on 11) The method of claim 11, wherein the training the first CNN according to the at least one first loss comprises training the first CNN according to the at least one first loss and the at least one second loss; wherein the at least one second loss is obtained by: transforming each of the plurality of original frames of the video by using a second CNN, the second CNN having been trained on an ImageNet dataset; transforming each of a plurality of the stylized frames by using the second CNN; and determining the at least one second loss according to an output feature vector of each of the plurality of the original frames at a first layer of the second CNN, and an output feature vector of each of the plurality of the stylized frames at a first layer of the second CNN. ### Claim 15 (depends on 14) The method of claim 14, wherein the at least one second loss comprises a content loss, and the content loss is obtained by: extracting a first feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to each of the plurality of original frames; extracting a second feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to a stylized frame corresponding to the original frame; and determining the content loss according to Euclidean distance between the first feature map and second feature map. ### Claim 16 (depends on 14) The method of claim 14, wherein the at least one second loss comprises a style loss, and the style loss is obtained by: determining a first Gram matrix according to a first feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to each of the plurality of original frames; determining a second Gram matrix according to a second feature map of an activation of a convolutional layer of the second CNN when the second CNN is applied to a stylized frame corresponding to the original frame; and determining the style loss according a difference between the first Gram matrix and second Gram matrix. ### Claim 17 (depends on 16) The method of claim 16, wherein determining the style loss according the difference between the first Gram matrix and second Gram matrix comprises: determining the style loss according a squared Frobenius norm of the difference between the first Gram matrix and second Gram matrix. ### Claim 18 (depends on 16) The method of claim 16, wherein training the first CNN according to the at least one first loss and the at least one second loss comprises: training the first CNN such that a weighted sum of the at least one first loss and the at least one second loss is minimized. ### Claim 19 (depends on 18) The method of claim 18, wherein training the first CNN such that a weighted sum of the at least one first loss and the at least one second loss is minimized comprises: training the first CNN based on a method which uses gradient to update network parameters of the first CNN, such that a weighted sum of the at least one first loss and the at least one second loss is minimized. ### Claim 20 A device for stylizing a video, comprising: a memory for storing instructions; and at least one processor configured to execute the instructions to perform operations of: stylizing a video by using a first convolutional neural network (CNN); wherein the first CNN has been trained according to at least one first loss which is determined according to a first original frame and second original frame of a plurality of original frames of the video and results of transforming, the second original frame being next to the first original frame, the transforming comprising: transforming each of a plurality of original frames of the video into a stylized frame by using a first CNN for stylizing.