Method and apparatus for applying deep learning techniques in video coding, restoration and video quality analysis (VQA)

A computer readable non-transitory medium storing instructions for optimizing an encoder for encoding video, said encoder comprised of a plurality of encoding steps, the instructions for-:

• receiving an unencoded video;
• performing an encoding step on the unencoded video at an encoder using a first encoding scheme to generate an encoded video, wherein one or more parameters of the first encoding scheme are selected based upon using a distortion measure indicating a level of distortion of the encoded video relative to the unencoded video;
• wherein the distortion measure corresponds to a combination of one or more spatial measures indicating a level of difference between a plurality of corresponding frames of the encoded video and the unencoded video, and one or more temporal measures indicating a level of difference between a first motion based upon a first frame difference between a plurality of frames of the unencoded video and a second motion based upon a second frame difference between a corresponding plurality of frames of the encoded video, wherein the spatial and temporal measures are computed at the frame or block level within the encoder prediction loop;
• and
• wherein the spatial measures and the temporal measures are combined using a machine learning method to determine a distortion measure used in selecting encoding parameters at the frame or block level within the encoder prediction loop.

A computer readable non-transitory medium storing instructions for determining an objective video quality measure of a video predictive of subjective human quality ratings, the instructions for:

• applying, at a video processing server comprising one or more processors and memory, a machine learning system trained to predict a subjective human quality rating for a given first video, by:
• determining a feature vector for the first video, by:
• generating a plurality of spatial measures of the first video selected from among measures capturing aesthetic measures of the first video;
• generating at least one temporal measure based upon one or more first motions associated with the first video, each first motion based upon analyzing at least a portion of two or more frames of the first video;
• and
• combining the generated plurality of spatial measures and at least one temporal measure to form the feature vector of the first video;
• processing the feature vector through the trained machine learning system to obtain an aggregate quality measure of the first video, the trained machine learning system configured to receive, for a given video, the plurality of spatial measures and the at least one temporal measure extracted from the given video, and output an aggregate quality measure of the video;
• wherein the plurality of spatial measures and at least one temporal measure used in the feature vector are selected from a pool of available features determined using a trained feature selector, wherein the feature selector is trained by computing said pool of features on a training database of videos, each video of the training database having a corresponding subjective score, and rank ordering the pool of features according to which features optimize the feature vector relative to the subjective scores according to a combination of correlation coefficients including at least one of a Pearson Correlation Coefficient or a Spearman Rank Correlation Coefficient.