System and method of video coding quantization and dynamic range control

A method for processing video residual data associated with a video block, comprising:

• transforming, via a processor, the video residual data to a first set of transform coefficients associated with the video block;
• determining, via the processor, based on a quantization step size parameter, a first scale factor of a set of scale factors to be applied for quantization, wherein the set of scale factors includes a first subset of scale factors;
• determining, via the processor, that a number of bit-shifts is a non-integer value;
• selecting, via the processor, a second scale factor of the set of scale factors on a condition that the number of bit-shifts for quantization is the non-integer value;
• quantizing, via the processor, the first set of transform coefficients to generate a first group of quantized coefficients, wherein the quantizing of the first set of transform coefficients includes applying the second scale factor of the set of scale factors,
• wherein on a condition that the second scale factor of the set of scale factors is associated with the first subset of scale factors, shifting the first group of quantized coefficients by a predetermined number of bits;
• entropy coding, via the processor, the first group of quantized coefficients or the shifted first group of quantized coefficients;
• and
• generating, via the processor, a bitstream including the entropy coded first group of quantized coefficients or the entropy coded shifted first group of quantized coefficients.

A method for processing video residual data associated with a video block, comprising:

• receiving, via a receiver, a bitstream including an entropy coded first group of quantized coefficients;
• determining, via a processor, based on a selected quantization step size parameter, a first inverse scale factor of a set of inverse scale factors to be applied for dequantization, wherein the set of inverse scale factors includes a first subset of inverse scale factors;
• determining, via the processor, that a number of bit-shifts is a non-integer value;
• selecting, via the processor, a second inverse scale factor of the set of inverse scale factors on a condition that the number of bit-shifts is the non-integer value;
• dequantizing, via the processor, the first group of quantized coefficients to generate a first set of dequantized transform coefficients, wherein the dequantizing of the first group of quantized coefficients includes applying the second inverse scale factor of the set of inverse scale factors,
• wherein on a condition that the second inverse scale factor of the set of inverse scale factors is associated with the first subset of inverse scale factors, shifting the first set of dequantized transform coefficients by a predetermined number of bits;
• and
• detransforming, via the processor, the first set of dequantized transform coefficients or the shifted first set of dequantized transform coefficients as a detransformed output to reconstruct the video residual data.