A large language model (LLM) ranking generation and reporting system, comprising:
- a user interface module;
- a communications module, connected to an internet, and configured to communicate with an LLM server through the internet;
- a memory module, storing a number n, wherein n is a positive integer greater than one;
- and
- a processing module, electrically connected to the user interface module, the communications module, and the memory module;
- wherein the processing module comprises an LLM processing unit and a math unit;
- when the processing module receives an input signal through the user interface module, the processing module executes the following steps:
- based on the input signal, executing a logic vector that contains a series of prompt commands in a looping logic structure for iteratively prompting an LLM on the LLM server about an internet entity for n times through the communications module, correspondingly receiving n batches of a plurality of answers from the LLM, and logging each of the batches of the answers as samples in the memory module;
- wherein the LLM processing unit accesses the LLM through the communications module to request for the answers from the LLM;
- wherein the answers comprise a first entity;
- calculating an appearance frequency of the first entity as a number of batches containing the first entity divided by n by the math unit, and calculating a confidence score for the first entity;
- displaying the first entity along with the appearance frequency of the first entity and the confidence score for the first entity through the user interface module;
- wherein each time when prompting the LLM on the LLM server about the internet entity, an AI prompt builder unit of the processing module queries the LLM to generate an LLM confidence level for having the first entity;
- wherein when logging each of the batches of the answers in the memory module, the LLM confidence levels for having the first entity are also logged;
- wherein the math unit calculates a confidence interval, and when calculating the confidence score for the first entity, the math unit executes the following steps:
- calculating an averaged LLM confidence level as a sum of the LLM confidence levels for having the first entity divided by n;
- normalizing the appearance frequency of the first entity as a normalized appearance frequency, normalizing the confidence interval as a normalized confidence interval, and normalizing the averaged LLM confidence level as a normalized averaged LLM confidence level;
- aggregating the normalized appearance frequency, the normalized confidence interval, and the normalized averaged LLM confidence level as the confidence score.