Feature-based hybrid video codec comparing compression efficiency of encodings

A digital processing system for processing video data having one or more video frames comprising:

• one or more computer processors executing an encoder;
• the encoder configured to use feature-based encoding to encode portions of the video frames and a first encoding process to encode other portions of the video frames by:
• detecting one or more instances of a candidate feature in one or more of the video frames, said candidate feature being a region of pels exhibiting encoding complexity relative to neighboring pels;
• using a motion compensated prediction process, segmenting the one or more instances of the candidate feature from non-features in the one or more video frames, the motion compensated prediction process selecting decoded video frames having features corresponding to the one or more instances of the candidate feature;
• determining positional information from the one or more decoded video frames;
• forming a feature-based model based on aggregating the candidate feature instances, the feature-based model including the positional information from the decoded video frames;
• normalizing the one or more candidate feature instances using the feature-based model, said normalizing using the positional information from the one or more decoded video frames as a positional prediction, resulting normalization being prediction of the one or more candidate feature instances in a subject video frame in the series;
• comparing the feature-based model to a video encoding resulting from the first video encoding process for the one or more instances of the candidate feature, and determining from the comparison which enables greater encoding compression;
• and
• using results of the comparing, applying feature-based encoding to portions of one or more of the video frames, and applying the first video encoding process to other portions of the one or more video frames.

A method of processing video data comprising:

• receiving video data having a series of video frames;
• detecting a candidate feature in one or more of the video frames, said candidate feature being a region of pels exhibiting encoding complexity relative to neighboring pels;
• segmenting the candidate feature from non-features in the video frame by employing reference frame processing used in a motion compensated prediction process;
• processing at least one or more portions of decoded video frames to identify potential matches in the one or more previously decoded video frames;
• determining that one or more of the decoded video frames include instances of the candidate feature;
• forming a representative feature model based on aggregating one or more of the candidate features by:
• aggregating the instances of the candidate feature into a set of instances of the candidate feature;
• and
• processing the candidate feature set to create a feature-based model, where the feature-based model includes one or more of a model of deformation variation or a model of appearance variation of the instances of the candidate feature, the appearance variation models being created by modeling pel variation of the instances of the candidate feature, the deformation variation models being created by modeling pel correspondence variation of the instances of the candidate feature;
• determining compression efficiency associated with using the feature-based model to model the candidate feature set;
• determining compression efficiency associated with using a first video encoding process to model the candidate feature set;
• comparing the feature-based model compression efficiency with the first video modeling compression efficiency, and determining which one is of greater compression value;
• encoding the video data using the feature-based models and the first video encoding process based on the compression efficiency.

A digital processing system for processing video data having one or more video frames comprising:

• one or more computer processors executing an encoder;
• the encoder configured to use feature-based encoding to encode portions of the video frames by:
• detecting a candidate feature in one or more of the video frames, said candidate feature being a region of pels exhibiting encoding complexity relative to neighboring pels;
• segmenting the candidate feature from non-features in the video frame by employing reference frame processing used in a motion compensated prediction process;
• processing at least the one or more portions of decoded video frames to identify potential matches of the candidate feature;
• determining that an amount of the portions of the decoded video frames include instances of the candidate feature;
• forming a representative feature model based on aggregating one or more of the candidate features by:
• aggregating the instances of the candidate feature into a set of instances of the candidate feature;
• and
• processing the candidate feature set to create a feature-based model, where the feature-based model includes a model of deformation variation and a model of appearance variation of the instances of the candidate feature, the appearance variation models being created by modeling pel variation of the instances of the candidate feature, the structural variation models being created by modeling pel correspondence variation of the instances of the candidate feature;
• determining compression efficiency associated with using the feature-based model to model the candidate feature set;
• determining compression efficiency associated with using the first video encoding process to model the candidate feature set;
• comparing the feature-based model compression efficiency with the video modeling compression efficiency of the first video encoding process;
• and
• encoding the video data using the feature-based models and the first video encoding process based on the compression efficiency.

A data processing system including:

• an encoder configured to process a series of video frames of video data, such that portions of the video data are encoded using a first encoding process and portions are encoded using a feature-based compression process by:
• processing a plurality of decoded video frames in the series to detect one or more instances of a candidate feature, said candidate feature being a region of pels exhibiting encoding complexity relative to neighboring pels;
• determining positional information for the detected instances of the candidate feature in the one or more decoded video frames;
• using a motion compensated prediction process to facilitate determining an instance of the candidate feature in a subject video frame in the series by initializing the motion compensated prediction process with the positional information for at least one of the detected instances of the candidate feature in the one or more decoded video frames;
• transforming one or more of the determined or the detected candidate feature instances;
• aggregating the transformed one or more of the determined or detected feature instances to create a feature-based encoding;
• and
• comparing compression efficiency of the feature-based encoding with an encoding from the first encoding process to determine which encoding process to apply.

A computer program product stored on a non-transitory computer useable medium configured to be executed by one or more processors, the computer program product being configured to cause the one or more processors to process video data by:

• encoding portions of the video data using a first encoding process and portions using a feature-based compression process by:
• processing a plurality of decoded video frames in the series to detect one or more instances of a candidate feature, said candidate feature being a region of pels exhibiting encoding complexity relative to neighboring pels;
• determining positional information for at least one of the detected instances of the candidate feature in the one or more decoded video frames;
• using a motion compensated prediction process to facilitate determining an instance of the candidate feature in a subject video frame in the series by initializing the motion compensated prediction process with the positional information for at least one of the detected instances of the candidate feature in the one or more decoded video frames;
• transforming one or more of the determined or the detected candidate feature instances;
• aggregating the transformed one or more of the determined or detected feature instances to create a feature-based encoding;
• and
• comparing compression efficiency of the feature-based encoding with an encoding from the first encoding process to determine which encoding process to apply.

A computer implemented method of processing a series of video frames of video data, comprising the computer implemented steps of:

• encoding portions of the video data using a first encoding process and portions using a feature-based compression process by:
• processing a plurality of decoded video frames in the series to detect one or more instances of a candidate feature, said candidate feature being a region of pels exhibiting encoding complexity relative to neighboring pels;
• determining positional information for at least one of the detected instances of the candidate feature in the one or more decoded video frames;
• using a motion compensated prediction process to facilitate determining an instance of the candidate feature in a subject video frame in the series by initializing the motion compensated prediction process with the positional information for at least one of the detected instances of the candidate feature in the one or more decoded video frames;
• transforming one or more of the determined or the detected candidate feature instances;
• aggregating one or more of the transformed candidate feature instances;
• using the aggregated and transformed feature instances to create a feature based model configured as a feature-based encoding;
• and
• comparing compression efficiency of the feature-based encoding with an encoding from the first encoding process to determine which encoding process to apply.