RDO-based small transform coefficients suppression algorithm for HEVC

A method for suppressing small transform coefficients using a rate-distortion optimization (RDO)-based algorithm, the method comprising:

• receiving a prediction residual associated with a transform unit (TU) of a video frame;
• processing the prediction residual to generate a plurality of coefficients and a processed output;
• generating a control signal based on the plurality of coefficients;
• and
• receiving the processed output and a zero-out data and selecting, as an output of a switch, one of the processed output or the zero-out data based on the control signal generated based on the plurality of coefficients comprising original coefficients, quantized coefficients, and inverse-quantized coefficients,
• wherein in the zero-out data a zero value is assigned to each of the plurality of coefficients of the TU.

A system for suppressing small transform coefficients using a rate-distortion optimization (RDO)-based algorithm, the system comprising:

• memory;
• and
• one or more processors coupled to the memory to execute one or more program modules to perform:
• processing a prediction residual to generate a plurality of coefficients and a processed output;
• generating a control signal based on the plurality of coefficients;
• and
• receiving the processed output and a zero-out data and selecting, as an output of a switch, one of the processed output or the zero-out data based on the control signal generated based on the plurality of coefficients comprising original coefficients, quantized coefficients, and inverse-quantized coefficients,
• wherein the prediction residual is associated with a transform unit (TU) of a video frame, and wherein the zero-out data comprises a zero value for each of the plurality of coefficients of the TU.