Method and system for calculating domain relevance scores for responses generated by large language models

A system for calculating domain relevance scores for responses generated by Large Language Models (LLMs), the system comprising:

• a processor;
• and
• a memory communicatively coupled to the processor, wherein the memory stores processor executable instructions, which, on execution, causes the processor to:
• receive a response generated by an LLM corresponding to a user query, wherein the user query is associated with a domain;
• split the response into a plurality of response chunks using a splitting technique;
• generate a plurality of response vector embeddings based on the plurality of response chunks using at least one sentence transformer;
• compute a plurality of cosine distances between the plurality of response vector embeddings and a corresponding plurality of training data vector embeddings, wherein the plurality of training data vector embeddings corresponds to domain-specific training data of the LLM;
• and
• calculate a domain relevance score corresponding to the response, based on a sum of the plurality of cosine distances and a number of the plurality of response chunks.

A non-transitory computer-readable medium storing computer-executable instructions for calculating domain relevance scores for responses generated by Large Language Models (LLMs), the stored instructions, when executed by a processor, cause the processor to perform operations comprises:

• receiving a response generated by an LLM corresponding to a user query, wherein the user query is associated with a domain;
• splitting the response into a plurality of response chunks using a splitting technique;
• generating a plurality of response vector embeddings based on the plurality of response chunks using at least one sentence transformer;
• computing a plurality of cosine distances between the plurality of response vector embeddings and a corresponding plurality of training data vector embeddings, wherein the plurality of training data vector embeddings corresponds to domain-specific training data of the LLM;
• and
• calculating a domain relevance score corresponding to the response, based on a sum of the plurality of cosine distances and a number of the plurality of response chunks.