Non-transitory computer-readable storage medium for storing operation program, operation method, and calculator

An operation method implemented by a computer, the operation method comprising:

• performing first learning with a high-precision data type in each of layers included in a learning model;
• calculating a number of bits to be used for quantization in each of the layers, based on a threshold value that corresponds to a first quantization error and a degree of attenuation by accumulation of quantization errors in a case where quantization is performed in the first learning;
• and
• repeatedly performing second learning that includes quantization in a data type based on the calculated number of bits for each of the layers until the second learning converges.

An operation apparatus comprising:

• a memory;
• and
• a processor coupled to the memory, the processor being configured to perform processing, the processing including:
• performing first learning with a high-precision data type in each of layers included in a learning model;
• calculating a number of bits to be used for quantization in each of the layers, based on a threshold value that corresponds to a first quantization error and a degree of attenuation by accumulation of quantization errors in a case where quantization is performed in the first learning;
• and
• repeatedly performing second learning that includes quantization in a data type based on the calculated number of bits for each of the layers until the second learning converges.