Video enhancement using a generator with filters of generative adversarial network

A computer-implemented method comprising:

• receiving a request to enhance one or more images of a streaming video file;
• receiving a plurality of lower-resolution images of the video to be at least partially enhanced using a trained neural network;
• enhancing a middle lower-resolution image of the plurality of lower-resolution images using a generator with filters of a generative adversary network according to the request by:
• temporally pre-processing the lower-resolution images by concatenating the lower-resolution images along a temporal dimension,
• temporally reducing the concatenated images,
• removing artifacts of the temporally reduced concatenated images at a first resolution to generate a first red, green, blue (RGB) image using an artifact removal layer and features of the first RGB image,
• processing features of the first RGB image at a second, higher resolution to generate a second RGB image,
• upsampling the features of the second RGB image,
• processing features of the upsampled second RGB image at a third, higher resolution to generate a third RGB image,
• upsampling the features of the third RGB image to generate a residual of the third RGB image,
• generating a filter from the features of the first RGB image,
• performing a product of the generated filter and the RGB image generated by the artifact removal layer, and
• summing the product with the residual of the third RGB image to generate an enhanced image;
• and
• outputting the enhanced image.

A system comprising:

• storage for a video file;
• an image enhancement service implemented by one or more electronic devices, the image enhancement service including instructions that upon execution cause the image enhancement service to:
• receive a request to enhance one or more images of the video file,
• receive a middle lower-resolution image of the video of a plurality of lower-resolution images to be at least partially enhanced using a trained neural network,
• enhance the middle lower-resolution image of the plurality of lower-resolution images using a generator with filters of a generative adversary network according to the request by:
• temporally pre-processing the lower-resolution images by concatenating the lower-resolution images along a temporal dimension,
• temporally reducing the concatenated images,
• removing artifacts of the temporally reduced concatenated images at a first resolution to generate a first red, green, blue (RGB) image using an artifact removal layer and features of the first RGB image,
• processing features of the first RGB image at a second, higher resolution to generate a second RGB image,
• upsampling the features of the second RGB image,
• processing features of the upsampled second RGB image at a third, higher resolution to generate a third RGB image,
• upsampling the features of the third RGB image to generate a residual of the third RGB image,
• generating a filter from the features of the first RGB image,
• performing a product of the generated filter and the RGB image generated by the artifact removal layer, and
• summing the product with the residual of the third RGB image to generate an enhanced image;
• and
• output the enhanced image.