Video codec architecture for next generation video

A video encoder comprising:

• an image buffer;
• a graphics processing unit communicatively coupled to the image buffer and configured to:
• segment a first video frame of a first type and a second video frame of a second type into a first plurality of tiles or super-fragments and a second plurality of tiles or super-fragments;
• partition the first plurality of tiles or super-fragments using a first partitioning technique and the second plurality of tiles or super-fragments using a second partitioning technique, wherein the first and second partitioning techniques are different;
• determine a selected prediction partitioning and an associated plurality of prediction partitions for the second video frame;
• generate a first reconstructed tile or super-fragment;
• generate deblock filtering parameters and a first final reconstructed tile or super-fragment based at least in part on the first reconstructed tile or super-fragment and the deblock filtering parameters;
• assemble the first final reconstructed tile or super-fragment and a second final reconstructed tile or super-fragment to generate a first decoded prediction reference picture;
• generate morphing characteristic parameters and a morphed prediction reference picture based at least in part on the morphing characteristic parameters and the first decoded reference picture;
• generate synthesizing characteristic parameters and a synthesized prediction reference picture based at least in part on the synthesizing characteristic parameters and the first decoded reference picture or a second decoded reference picture;
• determine a mode and a reference type for each of the plurality of prediction partitions;
• generate motion data associated with the plurality of prediction partitions based at least in part on one of the morphed prediction reference picture or the synthesized prediction reference picture;
• perform motion compensation based on the motion data and at least one of the morphed prediction reference picture or the synthesized prediction reference picture to generate a plurality of predicted partitions for the plurality of prediction partition;
• difference the plurality of predicted partitions with corresponding original pixel data to generate a corresponding plurality of prediction error data partitions;
• determine a subset of prediction error data partitions of the plurality of prediction error data partitions are required to be encoded;
• for the subset of prediction error data partitions that are required to be encoded:
• partition individual prediction error data partitions of the subset of prediction error data partitions into a selected coding partitioning and an associated plurality of coding partitions;
• perform content adaptive transforms on one or more first coding partitions and fixed transforms on one or more second coding partitions of the plurality of coding partitionings to generate transform coefficients;
• and
• quantize the transform coefficients to generate quantized transform coefficients;
• entropy encode frame data associated with the first type and the second type, mode data associated with the modes and the reference type, prediction partitioning indicators or prediction partitioning codewords associated with the selected prediction partitioning, coding partitioning indicators or coding partitioning codewords associated with the selected coding partitioning, the motion data, the quantized transform coefficients, the morphing characteristic parameters, the synthesizing characteristic parameters, and the deblock filtering parameters into a bitstream.

A decoder system comprising:

• an image buffer;
• a video decoder communicatively coupled to the image buffer, the video decoder configured to decode an encoded bitstream, wherein the video decoder is configured to:
• entropy decode the encoded bitstream to determine quantized transform coefficients, data associated with the selected content adaptive and fixed transforms, coding partitioning indicators or coding partitioning codewords, deblock filtering parameters, morphing characteristic parameters, synthesizing characteristic parameters, motion data, prediction partitioning indicators or prediction partitioning codewords,
• perform an inverse quantization based at least in part on the quantized transform coefficients to generate decoded transform coefficients;
• perform inverse transforms based at least in part on the decoded transform coefficients and the data associated with the selected content adaptive and fixed transforms to generate a plurality of decoded coding partitions;
• assemble the plurality of decoded coding partitions based at least in part on the coding partitioning indicators or coding partitioning codewords to generate a decoded prediction error data partition;
• generate at least one of a first decoded tile or super-fragment;
• apply deblock filtering to the first decoded tile or super-fragment based at least in part on the deblock filtering parameters to generate a first final decoded reconstructed tile or super-fragment;
• assemble the first final decoded reconstructed tile or super-fragment and a second final decoded reconstructed tile or super-fragment to generate a decoded reconstructed video frame;
• generate a morphed prediction reference picture based at least in part on the morphing characteristic parameters and the decoded reconstructed video frame;
• generate a synthesized prediction reference picture based at least in part on the synthesizing characteristic parameters and at least one of the reconstructed video frame or a second reconstructed video frame;
• perform motion compensation based at least in part on the decoder morphed prediction reference picture or the decoder synthesized prediction reference picture and the motion data to generate a decoded predicted partition;
• add the decoded predicted partition to the decoded prediction error data partition to generate a first decoded reconstructed partition;
• combine the first decoded reconstructed partition and a second decoded reconstructed partition based at least in part on the prediction partitioning indicators or prediction partitioning codewords to generate a third final decoded reconstructed tile or super-fragment;
• assemble the third final decoded reconstructed tile or super-fragment and a fourth final decoded reconstructed tile or super-fragment to generate a second decoded reconstructed video frame;
• and
• transmit the second decoded reconstructed video frame for presentment via a display device.

The decoder of system 14, further comprising:

• an antenna communicatively coupled to the antenna and configured to receive the encoded bitstream of video data;
• and
• a display device configured to present video frames,
• wherein the video decoder is further configured to:
• receive the bitstream;
• entropy decode the bitstream to determine frame data associated with a first type of the first decoded reconstructed video frame and a second type of the second decoded reconstructed video frame, mode data associated with a mode and a reference type for the prediction partition, enhanced prediction parameters, fusion filtering parameters or fusion improvement filtering parameters, deblock filtering or deblock and dither filtering parameters, and in-loop filtering or quality restoration filtering parameters, wherein to entropy decode comprises the video decoder being configured to implement at least one of an adaptive symbol-run variable length coding technique or an adaptive proxy variable length coding technique, and wherein to entropy decode the prediction partitioning indicators or the prediction partitioning codewords the video decoder being configured to implement a different entropy decoding technique than that used to entropy decode the quantized transform coefficients, wherein to perform motion compensation comprises the video decoder being configured to implement an adaptive motion filtering or an adaptive precision filtering based at least in part on the adaptive motion filtering parameters or adaptive precision filtering parameters;
• generate an enhanced decoded predicted partition based at least in part on the enhanced prediction parameters and the decoded predicted partition;
• determine whether to use the decoded enhanced predicted partition or the decoded predicted partition as a selected decoded predicted partition;
• determine whether to apply decoder fusion filtering or decoder fusion improvement filtering to the selected decoded predicted partition;
• when decoder fusion filtering or decoder fusion improvement filtering is to be applied to the selected decoded predicted partition:
• assemble the selected decoded predicted partition and a second selected decoded predicted partition to generate at least a portion of a decoded assembled picture;
• and
• fusion filter or fusion improvement filter the portion of the decoded assembled picture based at least in part on the fusion filtering parameters or the fusion improvement filtering parameters,
• wherein the coding partitioning indicators or coding partitioning codewords comprise bi-tree partitioning coding partitioning indicators or coding partitioning codewords,
• wherein the prediction partitioning indicators or prediction partitioning codewords comprise at least one of k-d tree partitioning or bi-tree partitioning prediction partitioning indicators or prediction partitioning codewords,
• wherein the inverse transforms comprises at least one of an inverse fixed transforms, an inverse discrete cosine transform, an inverse discrete cosine transform approximator, an inverse adaptive transform, or an inverse hybrid parametric Haar transform,
• wherein the morphed prediction reference picture comprises at least one of a gain modified prediction reference picture, a blur modified prediction reference picture, a dominant motion modified prediction reference picture, or a registration modified prediction reference picture and wherein the synthesized prediction reference picture comprises at least one of a super resolution prediction reference picture or a projection trajectory prediction reference picture,
• wherein generating the morphing characteristic parameters comprises at least one of generating the morphing characteristic parameters based at least in part on the second video frame and a previously decoded prediction reference picture or generating the morphing characteristic parameters based at least in part on the second video frame and a third original video frame, and
• wherein the motion data comprises a motion vector.

At least one non-transitory machine readable medium comprising a plurality of instructions that in response to being executed on a computing

• device, cause the computing device to:
• segment a first video frame of a first type and a second video frame of a second type into a first plurality of tiles or super-fragments and a second plurality of tiles or super-fragments;
• partition the first plurality of tiles or super-fragments using a first partitioning technique and the second plurality of tiles or super-fragments using a second partitioning technique, wherein the first and second partitioning techniques are different;
• determine a selected prediction partitioning and an associated plurality of prediction partitions for the second video frame;
• generate a first reconstructed tile or super-fragment;
• generate deblock filtering parameters and a first final reconstructed tile or super-fragment based at least in part on the first reconstructed tile or superfragment
• and the deblock filtering parameters;
• assemble the first final reconstructed tile or super-fragment and a second final reconstructed tile or super-fragment to generate a first decoded prediction reference picture;
• generate morphing characteristic parameters and a morphed prediction reference picture based at least in part on the morphing characteristic parameters and the first decoded reference picture;
• generate synthesizing characteristic parameters and a synthesized prediction reference picture based at least in part on the synthesizing characteristic parameters and the first decoded reference picture or a second decoded reference picture;
• determine a mode and a reference type for each of the plurality of prediction partitions;
• generate motion data associated with the plurality of prediction partitions based at least in part on one of the morphed prediction reference picture or the synthesized prediction reference picture;
• perform motion compensation based on the motion data and at least one of the morphed prediction reference picture or the synthesized prediction reference picture to generate a plurality of predicted partitions for the plurality of prediction partition;
• difference the plurality of predicted partitions with corresponding original pixel data to generate a corresponding plurality of prediction error data partitions;
• determine a subset of prediction error data partitions of the plurality of prediction error data partitions are required to be encoded;
• for the subset of prediction error data partitions that are required to be encoded:
• partition individual prediction error data partitions of the subset of prediction error data partitions into a selected coding partitioning and an associated plurality of coding partitions;
• perform content adaptive transforms on one or more first coding partitions and fixed transforms on one or more second coding partitions of the plurality of coding partitionings to generate transform coefficients;
• and
• quantize the transform coefficients to generate quantized transform coefficients;
• entropy encode frame data associated with the first type and the second type, mode data associated with the modes and the reference type, prediction partitioning indicators or prediction partitioning codewords associated with the selected prediction partitioning, coding partitioning indicators or coding partitioning codewords associated with the selected coding partitioning, the motion data, the quantized transform coefficients, the morphing characteristic parameters, the synthesizing characteristic parameters, and the deblock filtering parameters into a bitstream.