Electronic device, method, and non-transitory computer-readable medium for improving inference performance of a pre-trained language model by outputting middle-layer prediction result based on confidence level

A method performed by an electronic device comprising a memory storing a pre-trained language model and one or more instructions and a processor configured to execute the one or more instructions stored in the memory, the method performed by the processor comprising:

• sequentially passing input data through a plurality of transformer layers of the pre-trained language model and obtaining output data,
• wherein the obtaining of the output data comprises:
• monitoring an inference state of the pre-trained language model by calculating a probability distribution for prediction results received from each transformer layer in each of a plurality of middle layers, wherein each of the plurality of middle layers is connected to each rear end of the plurality of transformer layers;
• measuring a confidence level of the prediction results based on an entropy value calculated by using the probability distribution;
• managing a patience counter to accumulate a number of consecutive middle layers among the plurality of middle layers and in which the confidence level measured in each of the plurality of middle layers is less than a predefined value;
• and
• in response to the patience counter reaching a predefined number, dynamically control an inference path of the pre-trained language model by outputting a prediction result of first middle layer which is last middle layer among the consecutive middle layers as the output data by taking the first middle layer as an exit, and
• the method further comprises:
• optimizing a trade-off between inference speed and accuracy of the electronic device by variably adjusting at least one of the predefined number of the consecutive middle layers and the predefined value of the confidence level.

A non-transitory computer readable medium storing instructions comprising a code for causing a processor to execute:

• sequentially passing input data through a plurality of transformer layers of the pre-trained language model and obtaining output data,
• wherein obtaining of the output data comprises:
• monitoring an inference state of the pre-trained language model by calculating a probability distribution for prediction results received from each transformer layer in each of a plurality of middle layers, wherein each of the plurality of middle layers is connected to each rear end of the plurality of transformer layers and obtaining output data;
• measuring a confidence level of the prediction results based on an entropy value calculated by using the probability distribution;
• managing a patience counter to accumulate a number of consecutive middle layers among the plurality of middle layers and in which the confidence level measured in each of the plurality of middle layers is less than a predefined value;
• and
• in response to the patience counter reaching a predefined number, dynamically controlling an inference path of the pre-trained language model by outputting a prediction result of first middle layer which is last middle layer among the consecutive middle layers as the output data by taking the first middle layer as an exit, and
• the method further comprises:
• optimizing a trade-off between inference speed and accuracy of the processor by variably adjusting at least one of the predefined number of the consecutive middle layers and the predefined value of the confidence level.