Inter prediction mode-based image processing method and device therefor

An apparatus for processing an image based on an inter prediction mode, comprising:

• a memory configured to store data of the image;
• and
• a processor coupled to the memory,
• wherein the processor is configured for:
• generating a temporary candidate list by using a spatial neighbor block or temporal neighbor block of a current block;
• calculating a cost for each candidate in the temporary candidate list based on a difference between a template of the current block and a template of a reference block of a motion vector of the candidate;
• generating a final candidate list by rearranging candidates of the temporary candidate list based on the calculated cost;
• decoding a merge index indicating a specific candidate in the final candidate list;
• and
• generating a prediction block of the current block by using motion information of a candidate indicated by the merge index,
• wherein the processor is further configured for performing a normalization on the difference based on a width and a height of the current block when the current block is a non-square block.