Multi-view convolutional neural networks for video processing

A method for training a convolutional neural network (CNN) model for use in processing of input video data that can be partitioned into two or more clips, each clip comprising a number of T frames, wherein each frame comprises a frame height, a frame width and frame channel dimension Cin;

• wherein the CNN model comprises:
• a CNN encoder for encoding each clip into S encoded representations, wherein T≥S>1, wherein the CNN encoder comprises an encoder input layer associated with an input layer encoder convolution kernel comprising an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension Cout;
• and wherein the encoder kernel input channel dimension equals T times Cin and wherein the encoder kernel output channel dimension Cout equals S times a positive integer N;
• and
• a CNN decoder for decoding each encoded representation of each clip into a decoded frame of output video data;
• the method comprising:
• initializing network parameters of the CNN model;
• and
• training the CNN model based on one of:
• supervised learning, unsupervised learning, semi-supervised learning.

An apparatus for processing video data, comprising:

• at least one memory configured to store the video data;
• and
• at least one processor coupled to the at least one memory, the at least one processor configured to:
• partition input video data into two or more clips, each clip comprising a number of T frames, wherein each frame comprises a frame height, a frame width and a frame channel input dimension Cin;
• encode each clip into S encoded representations comprising a code height, a code width, and a code channel dimension, wherein T and S are integers with T≥S>1 and wherein encoding each clip into the S encoded representations comprises:
• concatenating all T frames of the clip into an input tensor along the frame channel dimension;
• and
• encoding the input tensor into the S encoded representations using a convolutional neural network (CNN) encoder;
• wherein the CNN encoder comprises an encoder input layer associated with an input layer encoder convolution kernel comprising an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension Cout;
• and wherein the encoder kernel input channel dimension equals T times Cin and wherein the encoder kernel output channel dimension Cout equals S times a positive integer N.

An apparatus for training a convolutional neural network (CNN) model for use in processing of input video data that can be partitioned into two or more clips, each clip comprising a number of T frames, wherein each frame comprises a frame height, a frame width and frame channel dimension Cin;

• wherein the CNN model comprises:
• a CNN encoder for encoding each clip into S encoded representations, wherein T≥S>1, wherein the CNN encoder comprises an encoder input layer associated with an input layer encoder convolution kernel comprising an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension Cout;
• and wherein the encoder kernel input channel dimension equals T times Cin and wherein the encoder kernel output channel dimension Cout equals S times a positive integer N;
• and
• a CNN decoder for decoding each encoded representation of each clip into a decoded frame of output video data;
• the apparatus comprising:
• at least one memory configured to store the input video data;
• and
• at least one processor coupled to the at least one memory, the at least one processor configured to:
• initialize network parameters of the CNN model;
• and
• train the CNN model based on one of supervised learning, unsupervised learning, semi-supervised learning.

A non-transitory computer-readable storage medium having computer executable code stored thereon, which when executed by one or more processors, causes the one or more processors to:

• partition input video data into two or more clips, each clip comprising a number of T frames, wherein each frame comprises a frame height, a frame width and a frame channel dimension Cin;
• encode each clip into S encoded representations comprising a code height, a code width, and a code channel dimension, wherein T and S are integers with T≥S>1 and wherein encoding each clip into the S encoded representations comprises:
• concatenating all T frames of the clip into an input tensor along the frame channel dimension;
• and
• encoding the input tensor into the S encoded representations using a convolutional neural network (CNN) encoder, wherein the CNN encoder comprises an encoder input layer associated with an input layer encoder convolution kernel comprising an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension Cout;
• and wherein the encoder kernel input channel dimension equals T times Cin and wherein the encoder kernel output channel dimension Cout equals S times a positive integer N.

A non-transitory computer-readable storage medium having stored thereon computer executable code for training a convolutional neural network (CNN) model for use in processing of input video data that can be partitioned into two or more clips, each clip comprising a number of T frames, wherein each frame comprises a frame height, a frame width and frame channel dimension Cin;

• wherein the CNN model comprises:
• a CNN encoder for encoding each clip into S encoded representations, wherein T≥S>1, wherein the CNN encoder comprises an encoder input layer associated with an input layer encoder convolution kernel comprising an encoder kernel height, an encoder kernel width, an encoder kernel input channel dimension, and an encoder kernel output channel dimension Cout;
• and wherein the encoder kernel input channel dimension equals T times Cin and wherein the encoder kernel output channel dimension Cout equals S times a positive integer N;
• and
• a CNN decoder for decoding each encoded representation of each clip into a decoded frame of output video data;
• the computer executable code, when executed by one or more processors, causes the one or more processors to:
• initialize network parameters of the CNN model;
• obtain labeled training video data;
• and
• train the CNN model based on the labeled training video data, wherein the training comprises:
• changing at least a subset of the initialized network parameters based on processing the labeled training video data with the CNN model.