# US 11,445,198 B2 · Multi-quality video super resolution with micro-structured masks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,198 B2 - Document ID: US11445198B2 - Status: Granted - Kind: B2 - Application number: 17/335,535 - Filed: 2021-06-01 - Published: 2022-03-31 - Granted: 2022-09-13 - Earliest claimed: 2020-09-29 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Wei Jiang; Wei Wang; Shan Liu - Web page: https://patentgrep.com/patent/US11445198B2 ## CPC - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - G06N 3/084: Backpropagation, e.g. using gradient descent - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method, computer program, and computer system is provided for encoding and decoding video data. Video data in a first resolution and a quality of the video data are received. One or more first weight coefficients of first layers a neural network and one or more binary masks for the one or more first weight coefficients are generated. One or more second weight coefficients of second layers of the neural network are generated based on the received quality of the video data. The video data is decoded in a second resolution by performing a convolution on the received video data in the first resolution using the generated one or more first weight coefficients that are masked by the generated one or more binary masks and using the generated one or more second weight coefficients. ## Claims ### Claim 1 A method of coding video data, executable by a processor, comprising: receiving video data in a first resolution and a quality, among a plurality of qualities, of the video data; generating one or more first weight coefficients of first layers of a neural network and one or more binary masks for the one or more first weight coefficients; generating one or more second weight coefficients of second layers of the neural network, based on the received quality of the video data; and decoding the video data in a second resolution by performing a convolution operation on the received video data in the first resolution, based on an inference computation using the generated one or more first weight coefficients that are masked by the generated one or more binary masks and using the generated one or more second weight coefficients, wherein a structure of the one or more binary masks corresponds to a matrix multiplication process based on the convolution operation, and wherein a respective set of the one or more binary masks is generated for each quality of the plurality of qualities, and in a set of the one or more binary masks corresponding to a given quality, each entry of the one or more binary masks is 1 or 0 indicating whether or not a corresponding weight entry in a set of the one or more first weight coefficients for the given quality participates in the inference computation for generating the video data in the second resolution for an input video with the given quality. ### Claim 2 (depends on 1) The method of claim 1, wherein the second resolution is higher than the first resolution. ### Claim 3 (depends on 1) The method of claim 1, wherein the one or more binary masks include block-wise micro-structures based on 3D reshaped weight tensor or a 2D reshaped weight matrix. ### Claim 4 (depends on 3) The method of claim 3, wherein all elements of the weight tensor or weight matrix have a same binary value of zero or 1. ### Claim 5 (depends on 1) The method of claim 1, wherein the neural network is trained by: determining a distortion loss based on a difference between the video data in the second resolution and a ground-truth video data in the first resolution; and updating the generated one or more first weight coefficients and the generated one or more second weight coefficients, to minimize the determined distortion loss. ### Claim 6 (depends on 5) The method of claim 5, wherein the neural network is further trained by: determining a pruning loss for the generated one or more first weight coefficients that are not masked by the generated one or more binary masks; pruning the one or more first weight coefficients not masked by the one or more binary masks, based on the determined pruning loss; and updating the pruned one or more first weight coefficients not masked by the one or more binary masks, to minimize the determined pruning loss. ### Claim 7 A computer system for video coding, the computer system comprising: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including: receiving code configured to cause the one or more computer processors to receive video data in a first resolution and a quality, among a plurality of qualities, of the video data; first generating code configured to cause the one or more computer processors to generate one or more first weight coefficients of first layers of a neural network and one or more binary masks for the one or more first weight coefficients; second generating code configured to cause the one or more computer processors to generate one or more second weight coefficients of second layers of the neural network, based on the received quality of the video data; and decoding code configured to cause the one or more computer processors to decode the video data in a second resolution by performing a convolution operation on the received video data in the first resolution, based on an inference computation using the generated one or more first weight coefficients that are masked by the generated one or more binary masks and using the generated one or more second weight coefficients, wherein a structure of the one or more binary masks corresponds to a matrix multiplication process based on the convolution operation, and wherein a respective set of the one or more binary masks is generated for each quality of the plurality of qualities, and in a set of the one or more binary masks corresponding to a given quality, each entry of the one or more binary masks is 1 or 0 indicating whether or not a corresponding weight entry in a set of the one or more first weight coefficients for the given quality participates in the inference computation for generating the video data in the second resolution for an input video with the given quality. ### Claim 8 (depends on 7) The computer system of claim 7, wherein the second resolution is higher than the first resolution. ### Claim 9 (depends on 7) The computer system of claim 7, wherein the one or more binary masks include block-wise micro-structures based on 3D reshaped weight tensor or a 2D reshaped weight matrix. ### Claim 10 (depends on 9) The computer system of claim 9, wherein all elements of the weight tensor or weight matrix have a same binary value of zero or 1. ### Claim 11 (depends on 7) The computer system of claim 7, wherein the neural network is trained by: determining a distortion loss based on a difference between the video data in the second resolution and a ground-truth video data in the first resolution; and updating the generated one or more first weight coefficients and the generated one or more second weight coefficients, to minimize the determined distortion loss. ### Claim 12 (depends on 11) The computer system of claim 11, wherein the neural network is further trained by: determining a pruning loss for the generated one or more first weight coefficients that are not masked by the generated one or more binary masks; pruning the one or more first weight coefficients not masked by the one or more binary masks, based on the determined pruning loss; and updating the pruned one or more first weight coefficients not masked by the one or more binary masks, to minimize the determined pruning loss. ### Claim 13 A non-transitory computer readable medium having stored thereon a computer program for video coding, the computer program configured to cause one or more computer processors to: receive video data in a first resolution and a quality, among a plurality of qualities, of the video data; generate one or more first weight coefficients of first layers of a neural network and one or more binary masks for the one or more first weight coefficients; generate one or more second weight coefficients of second layers of the neural network, based on the received quality of the video data; and decode the video data in a second resolution by performing a convolution operation on the received video data in the first resolution, based on an inference computation using the generated one or more first weight coefficients that are masked by the generated one or more binary masks and using the generated one or more second weight coefficients, wherein a structure of the one or more binary masks corresponds to a matrix multiplication process based on the convolution operation, and wherein a respective set of the one or more binary masks is generated for each quality of the plurality of qualities, and in a set of the one or more binary masks corresponding to a given quality, each entry of the one or more binary masks is 1 or 0 indicating whether or not a corresponding weight entry in a set of the one or more first weight coefficients for the given quality participates in the inference computation for generating the video data in the second resolution for an input video with the given quality. ### Claim 14 (depends on 13) The computer readable medium of claim 13, wherein the second resolution is higher than the first resolution. ### Claim 15 (depends on 13) The computer readable medium of claim 13, wherein the one or more binary masks include block-wise micro-structures based on 3D reshaped weight tensor or a 2D reshaped weight matrix. ### Claim 16 (depends on 13) The computer readable medium of claim 13, wherein the neural network is trained by: determining a distortion loss based on a difference between the video data in the second resolution and a ground-truth video data in the first resolution; and updating the generated one or more first weight coefficients and the generated one or more second weight coefficients, to minimize the determined distortion loss. ### Claim 17 (depends on 16) The computer readable medium of claim 16, wherein the neural network is further trained by: determining a pruning loss for the generated one or more first weight coefficients that are not masked by the generated one or more binary masks; pruning the one or more first weight coefficients not masked by the one or more binary masks, based on the determined pruning loss; and updating the pruned one or more first weight coefficients not masked by the one or more binary masks, to minimize the determined pruning loss.