# US 2025/0094138 A1 · Llm fine-tuning for code generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094138 A1
- Document ID: US20250094138A1
- Status: Published application
- Kind: A1
- Application number: 18/743,866
- Filed: 2024-06-14
- Published: 2025-03-20
- Earliest claimed: 2023-09-15
- Applicants: Oracle International Corporation
- Inventors: Yazhe HU; Mengqing GUO; Zheng WANG; Tao SHENG; Jun QIAN; Vinod MAMTANI
- Web page: https://patentgrep.com/patent/US20250094138A1

## CPC

- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- G06F 8/35: model driven
- G06F 8/36: Software reuse
- G06F 40/226: Validation
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/56: Natural language generation
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/091: Active learning
- H04L 51/04: Real-time or near real-time messaging, e.g. instant messaging [IM]
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems, methods, and other embodiments associated with automated fine-tuning of software code generation by large language models are described herein. In one embodiment, a method accesses a collection of software code samples that intermix sample code and human language description. The method generates prompts to an LLM to write code that performs as described by the human language description of the sample code. The method fine-tunes a large language model to generate software code based on a code generation loss function that evaluates code generated by the LLM in response to the prompts. The method generates an evaluation score for performance of the tuned large language model as a code generator based on code generation loss for second generated code. And, the method automatically signals that fine-tuning of the tuned large language is complete in response to the evaluation score satisfying a threshold.

## Claims

### Claim 1

One or more non-transitory computer-readable media that include stored thereon computer-executable instructions that when executed by at least a processor cause a computing system to:
access a collection of software code samples, wherein the software code samples includes intermixed sample code and human language description of the sample code;
generate prompts to an LLM to write code that performs as described by the human language description of the sample code;
fine-tune a large language model to generate software code based on a code generation loss function that evaluates code generated by the LLM in response to the prompts;
generate an evaluation score for performance of the tuned large language model as a code generator based on a value of the code generation loss function for second generated code; and
automatically determine to deploy the tuned large language model to a production environment for code generation in response to the evaluation score satisfying a threshold.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising computer-executable instructions that when executed by at least the processor cause the computing system to generate the code generation loss function to penalize one or more of dissimilarity of generated code with the sample code, incompleteness of the generated code, and inoperability of the generated code.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising computer-executable instructions that when executed by at least the processor cause the computing system to generate, as a component of the code generation loss function, a value of code matching loss that indicates an extent of dissimilarity between the sample code and the generated code.

### Claim 4 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the computer-executable instructions that when executed by at least the processor cause the computing system to generate a value of code matching loss further cause the computing system to:
generate a value of recall between the sample code and the generated code;
generate a value of precision between the sample code and the generated code; and
combine the values of recall and precision to produce the value of code matching loss.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising computer-executable instructions that when executed by at least the processor cause the computing system to generate, as a component of the code generation loss function, a value of non-linear code completeness loss that indicates an extent to which the generated code fails to produce expected output on a line-by-line basis.

### Claim 6 (depends on 5)

The non-transitory computer-readable medium of claim 5, wherein the computer-executable instructions that when executed by at least the processor cause the computing system to generate a value of non-linear code completeness loss further cause the computing system to:
access a test case associated with the sample code;
record line-by-line outputs of executing the sample code and the generated code on the test case;
compare the outputs of corresponding individual lines of the sample code and the generated code; and
where the outputs differ for a corresponding individual line, add to the value of non-linear code completeness loss an amount based on a line number of the corresponding individual line.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising computer-executable instructions that when executed by at least the processor cause the computing system to generate, as a component of the code generation loss function, a value of unit test passing loss that indicates an extent to which the generated code fails to produce output consistent with the sample code.

### Claim 8 (depends on 7)

The non-transitory computer-readable medium of claim 7, wherein the computer-executable instructions that when executed by at least the processor cause the computing system to generate a value of unit test passing loss further cause the computing system to:
access a test case associated with the sample code;
execute the sample code and the generated code on the test case;
compare results of the test case executed by the sample code and executed by the generated code; and
where the results differ, set the value of the unit test passing loss to indicate that unit test has failed.

### Claim 9 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the computer-executable instructions that when executed by at least the processor cause the computing system to generate prompts to an LLM to write code that performs as described by the human language description of the sample code further cause the computing system to:
automatically detect a coding language from the sample code; and
include the detected coding language in the prompts.

### Claim 10

A computing system, comprising:
at least one processor connected to at least one memory;
one or more non-transitory computer-readable media including computer-executable instructions stored thereon that when executed by at least the processor cause the computing system to:
access a collection of software code samples, wherein the software code samples includes intermixed sample code and human language description of the sample code;
generate prompts to an LLM to write code that performs as described by the human language description of the sample code;
fine-tune a large language model to generate software code based on a code generation loss function that evaluates code generated by the LLM in response to the prompts;
generate an evaluation score for performance of the tuned large language model as a code generator based on a value of the code generation loss function for second generated code; and
automatically determine to deploy the tuned large language model to a production environment for code generation in response to the evaluation score satisfying a threshold.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the computer-executable instructions further include instructions that when executed by at least the processor cause the computing system to generate, as a component of the code generation loss function, a value of code matching loss that indicates an extent of dissimilarity between the sample code and the generated code.

### Claim 12 (depends on 10)

The computing system of claim 10, wherein the computer-executable instructions further include instructions that when executed by at least the processor cause the computing system to generate, as a component of the code generation loss function, a value of non-linear code completeness loss that indicates an extent to which the generated code fails to produce expected output on a line-by-line basis.

### Claim 13 (depends on 10)

The computing system of claim 10, wherein the computer-executable instructions further include instructions that when executed by at least the processor cause the computing system to generate, as a component of the code generation loss function, a value of unit test passing loss that indicates an extent to which the generated code fails to produce output consistent with the sample code.

### Claim 14 (depends on 10)

The computing system of claim 10, wherein the computer-executable instructions further include instructions that when executed by at least the processor cause the computing system to automatically generate one or more test cases based on Monte Carlo simulation of inputs to the sample code.

### Claim 15

A computer-implemented method, comprising:
accessing a collection of software code samples, wherein the software code samples includes intermixed sample code and human language description of the sample code;
generating prompts to an LLM to write code that performs as described by the human language description of the sample code;
fine-tuning a large language model to generate software code based on a code generation loss function that evaluates code generated by the LLM in response to the prompts;
generating an evaluation score for performance of the tuned large language model as a code generator based on a value of the code generation loss function for second generated code; and
automatically signal that the fine tuning of the tuned large language model is complete in response to the evaluation score satisfying a threshold.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, further comprising:
generating a value of recall between the sample code and the generated code;
generating a value of precision between the sample code and the generated code; and
combining the values of recall and precision to produce a value of code matching loss, wherein the value of code matching loss is a component of the code generation loss function.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15, further comprising:
generating a value of recall between the sample code and the generated code;
generating a value of precision between the sample code and the generated code; and
combining the values of recall and precision to produce a value of code matching loss, wherein the value of code matching loss is a component of the code generation loss function.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, further comprising:
accessing a test case associated with the sample code;
recording line-by-line outputs of executing the sample code and the generated code on the test case;
comparing the outputs of corresponding individual lines of the sample code and the generated code;
where the outputs differ for a corresponding individual line, assign a loss amount based on a line number of the corresponding individual line; and
sum the assigned loss amounts for the individual lines to produce a value of non-linear code completeness loss, wherein the value of non-linear code completeness loss is a component of the code generation loss function.

### Claim 19 (depends on 15)

The computer-implemented method of claim 15, further comprising:
accessing a set of test cases automatically generated for inputs to the sample code;
for test cases in the set of test cases,
executing the sample code and the generated code on the test case,
comparing results of the executing the sample code on the test case and the executing the generated code on the test case,
where the results differ, incrementing a tally of differing results; and
determining a ratio of the tally of differing results to a count of the test cases to produce a value of unit test passing loss, wherein the value of unit test passing loss is a component of the code generation loss function.

### Claim 20 (depends on 15)

The computer-implemented method of claim 15, further comprising accessing one or more test cases.
