Mixed-precision multiply-and-accumulation tree structure to maximize memory bandwidth usage for computational acceleration of generative large language model

A multiply-and-accumulation (MAC) tree-based operator comprising:

• a plurality of floating-point (FP) multipliers connected in parallel and configured to process a multiplication operation on data delivered from an external memory;
• a plurality of first converters configured to convert output of each of the plurality of FP multipliers from floating point to fixed point;
• a fixed-point (FXP) adder tree connected to the plurality of first converters and configured to process summation of multiplication results of the plurality of FP multipliers;
• an FXP accumulator configured to accumulate output of the FXP adder tree;
• and
• a second converter configured to convert output of the FXP accumulator from the fixed point to the floating point,
• wherein each of the plurality of FP multipliers is configured to simultaneously process first multiplication between a first operand with first bit precision and a (2-1)-th operand with second bit precision and second multiplication between the first operand and a (2-2)-th operand with the second bit precision in response to a high-performance mode being selected and to simultaneously compute a first result value of the first multiplication and a second result value of the second multiplication.

An operating method of a multiply-and-accumulation (MAC) tree-based operator, wherein the MAC tree-based operator comprises a plurality of floating-point (FP) multipliers connected in parallel, a plurality of first converters connected to the plurality of FP multipliers, a fixed-point (FXP) adder tree, an FXP accumulator, and a second converter, and the method comprises:

• processing, using the plurality of FP multipliers, a multiplication operation on data delivered from an external memory;
• converting, using the plurality of first converters, a result of multiplication operation of each of the plurality of FP multipliers from floating point to fixed point;
• processing, using the FXP adder tree, summation of the converted result of the plurality of FP multipliers;
• accumulating, using the FXP accumulator, output of the FXP adder tree;
• and
• converting, using the second converter, output of the FXP accumulator from the fixed point to the floating point, and
• the MAC tree-based operator corresponds to one of a plurality of MAC tree-based operators included in a hardware accelerator for acceleration of an artificial intelligence (AI) model, and
• at least one of the number of the plurality of MAC tree-based operators included in the hardware accelerator and the number of the plurality of FP multipliers included in the MAC tree-based operator is determined based on a memory bandwidth provided for the hardware accelerator.