Method and system of testing a fine-tuned llm for domain specific code generation

A system for testing a fine-tuned large language model (LLM), comprising:

• a processor;
• and
• a memory communicably coupled to the processor, wherein the memory stores processor-executable instructions, which when executed by the processor, cause the processor to:
• receive a test dataset corresponding to a domain from a code repository,
• wherein the test dataset comprises a set of test function and a test code corresponding to each test function of the set of test functions;
• for each of the set of test functions:
• determine an LLM generated problem statement based on the corresponding test code using the fine-tuned LLM,
• wherein the fine-tuned LLM is fine-tuned based on a training dataset corresponding to the domain;
• and
• prompt the fine-tuned LLM based on the LLM generated problem statement to determine an LLM generated code for a corresponding test function;
• and
• determine an accuracy level of the fine-tuned LLM based on a percentage match between the LLM generated code with the test code for each of the set of test functions.

A non-transitory computer-readable medium storing computer-executable instructions for testing a fine-tuned large language model (LLM), the stored instructions, when executed by a processor, cause the processor to perform operations comprising:

• receiving a test dataset corresponding to a domain from a code repository,
• wherein the test dataset comprises a set of test functions and a test code corresponding to each test function of the set of test functions;
• for each of the set of test functions:
• determining an LLM generated problem statement, based on the corresponding test code using the fine-tuned LLM,
• wherein the fine-tuned LLM is fine-tuned based on a training dataset corresponding to the domain;
• and
• prompting the fine-tuned LLM, based on the LLM generated problem statement to determine an LLM generated code for a corresponding test function;
• and
• determining an accuracy level of the fine-tuned LLM, based on a percentage match between the LLM generated code with the test code for each of the set of test functions.