# US 2026/0087367 A1 · Automatic confidence based detection for fine-tuning of large language models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087367 A1
- Document ID: US20260087367A1
- Status: Published application
- Kind: A1
- Application number: 18/898,472
- Filed: 2024-09-26
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Kun HU; Jiaxin ZHANG
- Web page: https://patentgrep.com/patent/US20260087367A1

## CPC

- G06F 40/30: Semantic analysis
- G06N 3/096: Transfer learning
- G06N 3/08: Learning methods
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are disclosed for fine-tuning a large language model (LLM). In particular, a system calculates when to again fine-tune an LLM based on a newly obtained dataset. The system identifies whether the LLM's use of the new dataset causes more than a tolerable difference in the outputs to a defined set of inputs. As such, the system uses the LLM to generate a new set of outputs and corresponding set of confidences for the defined set of inputs based on the new dataset, and the system generates a new distribution of the new set of confidences. The system also stores a previous distribution of a previous set of confidences in a previous set of outputs to the defined set of inputs based on a previous dataset, and the system calculates a difference between the new distribution and the previous distribution. Fine-tuning the LLM is based on the difference.

## Claims

### Claim 1

A computer-implemented method for fine-tuning a large language model (LLM), the method comprising:
providing a set of inputs to a fine-tuned LLM;
generating, by the fine-tuned LLM, a first set of outputs based on the fine-tuned LLM analyzing a first dataset to respond to the set of inputs;
generating, by the fine-tuned LLM, a first set of confidences for the first set of outputs, wherein each confidence of the first set of confidences corresponds to an output of the first set of outputs;
obtaining a second dataset;
generating, by the fine-tuned LLM, a second set of outputs based on the fine-tuned LLM analyzing the second dataset to respond to the set of inputs;
generating, by the fine-tuned LLM, a second set of confidences for the second set of outputs, wherein each confidence of the second set of confidences corresponds to an output of the second set of outputs;
calculating a difference between a first distribution of the first set of confidences and a second distribution of the second set of confidences;
determining whether to fine-tune the fine-tuned LLM using the second dataset, wherein the determination is based on the difference; and
fine-tuning the fine-tuned LLM using the second dataset based on the determination.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
obtaining a pretrained LLM;
obtaining the first dataset; and
fine-tuning the pretrained LLM using the first dataset to generate the fine-tuned LLM.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
calculating the difference between the first distribution and the second distribution includes performing a Kolmogorov-Smirnov (K-S) test between the first distribution and the second distribution; and
determining whether to fine-tune the fine-tuned LLM using the second dataset includes identifying whether the first distribution and the second distribution are from a same distribution based on the difference.

### Claim 4 (depends on 3)

The method of claim 3, wherein identifying whether the first distribution and the second distribution are from the same distribution includes comparing the difference to a difference threshold based on a hyperparameter alpha defined for the K-S test.

### Claim 5 (depends on 4)

The method of claim 4, wherein the hyperparameter alpha is defined as 0.05.

### Claim 6 (depends on 3)

The method of claim 3, wherein determining whether to fine-tune the fine-tuned LLM using the second dataset further includes calculating one or more relative entropies based on the first distribution and the second distribution in response to identifying that the first distribution and the second distribution are not from the same distribution, wherein the determination is further based on the one or more relative entropies.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
generating, by the pretrained LLM before fine-tuning using the first dataset, an initial set of outputs based on the pretrained LLM analyzing the first dataset to respond to the set of inputs; and
generating, by the pretrained LLM before fine-tuning using the first dataset, an initial set of confidences for the initial set of outputs, wherein each confidence of the initial set of confidences corresponds to an output of the initial set of outputs and the initial set of confidences form an initial distribution.

### Claim 8 (depends on 7)

The method of claim 7, wherein calculating the one or more relative entropies includes:
calculating a first relative entropy from the initial distribution to the second distribution; and
calculating a second relative entropy from the initial distribution to the first distribution.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining whether to fine-tune the fine-tuned LLM using the second dataset further includes:
generating a divergence metric from the first relative entropy and the second relative entropy; and
comparing the divergence metric to a divergence threshold, wherein the determination is based on the comparison.

### Claim 10 (depends on 9)

The method of claim 9, wherein the divergence metric equals the first relative entropy divided by the second relative entropy.

### Claim 11

A computing system for fine-tuning a large language model (LLM), the system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, causes the system to perform operations comprising:
providing a set of inputs to a fine-tuned LLM;
generating, by the fine-tuned LLM, a first set of outputs based on the fine-tuned LLM analyzing a first dataset to respond to the set of inputs;
generating, by the fine-tuned LLM, a first set of confidences for the first set of outputs, wherein each confidence of the first set of confidences corresponds to an output of the first set of outputs;
obtaining a second dataset;
generating, by the fine-tuned LLM, a second set of outputs based on the fine-tuned LLM analyzing the second dataset to respond to the set of inputs;
generating, by the fine-tuned LLM, a second set of confidences for the second set of outputs, wherein each confidence of the second set of confidences corresponds to an output of the second set of outputs;
calculating a difference between a first distribution of the first set of confidences and a second distribution of the second set of confidences;
determining whether to fine-tune the fine-tuned LLM using the second dataset, wherein the determination is based on the difference; and
fine-tuning the fine-tuned LLM using the second dataset based on the determination.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise:
obtaining a pretrained LLM;
obtaining the first dataset; and
fine-tuning the pretrained LLM using the first dataset to generate the fine-tuned LLM.

### Claim 13 (depends on 12)

The system of claim 12, wherein:
calculating the difference between the first distribution and the second distribution includes performing a Kolmogorov-Smirnov (K-S) test between the first distribution and the second distribution; and
determining whether to fine-tune the fine-tuned LLM using the second dataset includes identifying whether the first distribution and the second distribution are from a same distribution based on the difference.

### Claim 14 (depends on 13)

The system of claim 13, wherein identifying whether the first distribution and the second distribution are from the same distribution includes comparing the difference to a difference threshold based on a hyperparameter alpha defined for the K-S test.

### Claim 15 (depends on 14)

The system of claim 14, wherein the hyperparameter alpha is defined as 0.05.

### Claim 16 (depends on 13)

The system of claim 13, wherein determining whether to fine-tune the fine-tuned LLM using the second dataset further includes calculating one or more relative entropies based on the first distribution and the second distribution in response to identifying that the first distribution and the second distribution are not from the same distribution, wherein the determination is further based on the one or more relative entropies.

### Claim 17 (depends on 16)

The system of claim 16, wherein the operations further comprise:
generating, by the pretrained LLM before fine-tuning using the first dataset, an initial set of outputs based on the pretrained LLM analyzing the first dataset to respond to the set of inputs; and
generating, by the pretrained LLM before fine-tuning using the first dataset, an initial set of confidences for the initial set of outputs, wherein each confidence of the initial set of confidences corresponds to an output of the initial set of outputs and the initial set of confidences form an initial distribution.

### Claim 18 (depends on 17)

The system of claim 17, wherein calculating the one or more relative entropies includes:
calculating a first relative entropy from the initial distribution to the second distribution; and
calculating a second relative entropy from the initial distribution to the first distribution.

### Claim 19 (depends on 18)

The system of claim 18, wherein determining whether to fine-tune the fine-tuned LLM using the second dataset further includes:
generating a divergence metric from the first relative entropy and the second relative entropy; and
comparing the divergence metric to a divergence threshold, wherein the determination is based on the comparison.

### Claim 20 (depends on 19)

The system of claim 19, wherein the divergence metric equals the first relative entropy divided by the second relative entropy.
