Extended low-frequency non-separable transform (LFNST) designs with worst-case complexity handling

A device for decoding video data, the device comprising:

• a memory configured to store video data;
• one or more processors implemented in circuitry and configured to:
• determine a number of allowed non-zero coefficients for a block of reconstructed values according to the equation NZ=floor((WCM×nTbW×nTbH)/(TD1×TD2)), wherein nTbH represents a height of the block, nTbW represents a width of the block, NZ equals the number of allowed non-zero coefficients, TD1 represents a first dimension of an inverse low-frequency non-separable transform (LFNST), TD2 represents a second dimension of the LFNST, and WCM represents a constant value;
• obtain a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients, wherein a number of coefficients in the first subset of dequantized coefficients is equal to the number of allowed non-zero coefficients for the block of video data;
• apply an inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients;
• and
• apply an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.