Front-end architecture for neural network based video coding

An apparatus for processing video data, comprising:

• a memory;
• and
• a processor coupled to the memory and configured to:
• generate, using a first convolutional layer of an encoder sub-network of a neural network system, output values associated with a luminance channel of a frame;
• generate, using a second convolutional layer of the encoder sub-network, output values associated with at least one chrominance channel of the frame;
• generate, using a third convolutional layer based on the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame, a combined representation of the frame;
• and
• generate encoded video data based on the combined representation of the frame.

A method of processing video data, the method comprising:

• obtaining an encoded frame;
• separating, by a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame;
• generating, by a second convolutional layer of the decoder sub-network of a neural network system, reconstructed output values associated with the luminance channel of the encoded frame;
• generating, by a third convolutional layer of the decoder sub-network, reconstructed output values associated with the at least one chrominance channel of the encoded frame;
• and
• generating an output frame including the reconstructed output values associated with the luminance channel and the reconstructed output values associated with the at least one chrominance channel.

An apparatus for processing video data, comprising:

• a memory;
• and
• a processor coupled to the memory and configured to:
• obtain an encoded frame;
• separate, using a first convolutional layer of a decoder sub-network, a luminance channel of the encoded frame from at least one chrominance channel of the encoded frame;
• generate, using a second convolutional layer of the decoder sub-network of a neural network system, reconstructed output values associated with the luminance channel of the encoded frame;
• generate, using a third convolutional layer of the decoder sub-network, reconstructed output values associated with the at least one chrominance channel of the encoded frame;
• and
• generate an output frame including the reconstructed output values associated with the luminance channel and the reconstructed output values associated with the at least one chrominance channel.