Apparatus and method for performing rate-distortion optimization based on Hadamard-quantization cost

An encoding apparatus, comprising:

• a hardware processor configured to
• select an encoding mode to be used to encode a target block from among multiple modes;
• and
• perform computation for rate-distortion optimization in the encoding mode,
• wherein the hardware processor calculates a rate-distortion cost of at least one of the multiple modes in relation to encoding of the target block, and selects the encoding mode from among the multiple modes, based on the cost,
• the multiple modes comprise a skip mode, a merge mode, an inter mode, and an intra mode,
• the hardware processor sets modes such that prediction in the skip mode and in the merge mode is performed, and such that prediction of the target block in the inter mode is performed while prediction of the target block in the intra mode is omitted if a level of a Hadamard value in the inter mode is in a high cost state, and
• the high cost state indicates a state in which the level of the Hadamard value in the inter mode has a value that causes prediction to be performed at a subsequent depth of the target block.

An encoding method, comprising:

• selecting an encoding mode to be used to encode a target block from among multiple modes;
• and
• performing computation for rate-distortion optimization in the selected encoding mode,
• wherein selecting the encoding mode comprises:
• calculating a rate-distortion cost of at least one of the multiple modes in relation to encoding of the target block;
• and
• selecting the encoding mode from among the multiple modes, based on the cost,
• the multiple modes comprise an inter mode, and
• whether to perform prediction in the inter mode upon encoding the target block is determined based on both a level of a Hadamard value in the inter mode and a first threshold function.