# US 2026/0087368 A1 · Fine-tuning language models for reasoning with counterfactual feedback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087368 A1
- Document ID: US20260087368A1
- Status: Published application
- Kind: A1
- Application number: 19/075,729
- Filed: 2025-03-10
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Javier GONZÁLEZ HERNÁNDEZ; Aditya Vithal NORI; Xinnuo XU; Jacqueline MAASCH; Alihan HÜYÜK
- Web page: https://patentgrep.com/patent/US20260087368A1

## CPC

- G06N 3/096: Transfer learning
- G06F 21/554: {involving event detection and direct action}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 2221/033: Test or assess software
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Example solutions for fine-tuning a language model include: generating a dataset that includes a plurality of paired samples, each paired sample of the plurality of paired samples includes (i) a factual question and a true outcome for that factual question and (ii) a counterfactual question and a true outcome for that counterfactual question; submitting a factual query to an answer model, the factual query including the factual question and the true outcome of the factual question, the answer model generating a factual answer in response to the factual query; submitting a counterfactual query to the answer model, the counterfactual query including the counterfactual question and the true outcome of the counterfactual question, the answer model generating a counterfactual answer in response to the counterfactual query; and performing fine-tuning on a target model using at least the factual question paired with factual answer and the counterfactual question paired with counterfactual answer.

## Claims

### Claim 1

A computerized method for fine-tuning a language model, the method comprising:
generating a dataset that includes a plurality of paired samples, each paired sample of the plurality of paired samples includes (i) a factual question and a true outcome for that factual question and (ii) a counterfactual question and a true outcome for that counterfactual question;
submitting a factual query to an answer model, the factual query including the factual question and the true outcome of the factual question;
receiving a factual answer from the answer model in response to the factual query;
submitting a counterfactual query to the answer model, the counterfactual query including the counterfactual question and the true outcome of the counterfactual question;
receiving a counterfactual answer from the answer model in response to the counterfactual query; and
performing fine-tuning on a target model using at least the factual question paired with the factual answer and the counterfactual question paired with the counterfactual answer.

### Claim 2 (depends on 1)

The computerized method of claim 1, further comprising:
submitting a cybersecurity query to the target model, the cybersecurity query including a security log from a computing device and a prompt instructing analysis of the security log to identify suspicious activity, thereby causing the target model to generate an output identifying at least one anomalous event from the security log.

### Claim 3 (depends on 1)

The computerized method of claim 1, wherein performing fine-tuning on the target model further includes performing supervised fine-tuning on the target model, wherein the factual question and the factual answer represent a first input-output pair and the counterfactual question and the counterfactual answer represent a second input-output pair.

### Claim 4 (depends on 1)

The computerized method of claim 1, further comprising:
generating the factual query by concatenating the factual question with the true outcome of the factual question, thereby causing the true outcome of the factual question to appear at the end of the factual query; and
generating the counterfactual query by concatenating the counterfactual question with the true outcome of the counterfactual question, thereby causing the true outcome of the counterfactual question to appear at the end of the counterfactual query.

### Claim 5 (depends on 1)

The computerized method of claim 1, wherein the factual answer includes the true outcome of the factual question, wherein the counterfactual answer includes the true outcome of the counterfactual question.

### Claim 6 (depends on 1)

The computerized method of claim 1, wherein the counterfactual question is a reformulation of the factual question where a premise and assumption included in the factual question are altered in the counterfactual question such as to contradict the factual question.

### Claim 7 (depends on 1)

The computerized method of claim 1, further comprising:
generating the factual question using a factual question template and inserting a first parameter into the factual question template; and
generating the counterfactual question using a counterfactual question template and inserting said first parameter into the counterfactual question template.

### Claim 8

A system for fine-tuning generative artificial intelligence (GAI) models, the system comprising:
a processor; and
a memory comprising computer-readable instructions, the processor, the memory and the computer-readable instructions configured to cause the processor to:
generate a dataset that includes a plurality of paired samples, each paired sample of the plurality of paired samples includes (i) a factual question and a true outcome for that factual question and (ii) a counterfactual question and a true outcome for that counterfactual question;
submit a factual query to an answer model, the factual query including the factual question and the true outcome of the factual question, thereby causing the answer model to generate a factual answer in response to the factual query;
submit a counterfactual query to the answer model, the counterfactual query including the counterfactual question and the true outcome of the counterfactual question, thereby causing the answer model to generate a counterfactual answer in response to the counterfactual query; and
perform fine-tuning on a target model using at least the factual question paired with the factual answer and the counterfactual query paired with the counterfactual answer.

### Claim 9 (depends on 8)

The system of claim 8, wherein performing fine-tuning on the target model further includes performing supervised fine-tuning on the target model, wherein the factual question and the factual answer represent a first input-output pair and the counterfactual question and the counterfactual answer represent a second input-output pair.

### Claim 10 (depends on 8)

The system of claim 8, wherein the processor, the memory and the computer-readable instructions are further configured to cause the processor to:
generate the factual query by concatenating the factual question with the true outcome of the factual question, thereby causing the true outcome of the factual question to appear at the end of the factual query; and
generate the counterfactual query by concatenating the counterfactual question with the true outcome of the counterfactual question, thereby causing the true outcome of the counterfactual question to appear at the end of the counterfactual query.

### Claim 11 (depends on 8)

The system of claim 8, wherein the factual answer includes the true outcome of the factual question, wherein the counterfactual answer includes the true outcome of the counterfactual question.

### Claim 12 (depends on 8)

The system of claim 8, wherein the counterfactual question is a reformulation of the factual question where a premise and assumption included in the factual question are altered in the counterfactual question such as to contradict the factual question.

### Claim 13 (depends on 8)

The system of claim 8, wherein the processor, the memory and the computer-readable instructions are further configured to cause the processor to:
generate the factual question using a factual question template and inserting a first parameter into the factual question template; and
generate the counterfactual question using a counterfactual question template and inserting said first parameter into the counterfactual question template.

### Claim 14 (depends on 8)

The system of claim 8, wherein the answer model is a language model configured to generate outputs in a natural language, wherein the factual answer and the counterfactual answer are in the natural language, wherein the factual answer begins with the true outcome of the factual question, wherein the counterfactual answer begins with the true outcome of the counterfactual question.

### Claim 15

A computer storage medium having computer-executable instructions that, upon execution by a processor of a computer, cause the processor to at least:
generate a dataset that includes a plurality of paired samples, each paired sample of the plurality of paired samples includes a factual question and a counterfactual question;
submit a plurality of factual queries to a target model, each factual query of the plurality of factual queries including the factual question and a different sampling temperature, thereby causing the target model to vary randomness of output;
receiving a plurality of factual answers from the target model in response to the submission of the plurality of factual queries;
submit a plurality of counterfactual queries to the answer model, each counterfactual query of the plurality of counterfactual queries including the counterfactual question and a different sampling temperature, thereby causing the target model to vary randomness of output;
receive a plurality of counterfactual answers from the target model in response to the submission of the plurality of counterfactual queries;
identify a preferred factual answer from the plurality of factual answers, the remaining factual answers of the plurality of factual answers being a disfavored factual answer;
identify a preferred counterfactual answer from the plurality of counterfactual answers, the remaining counterfactual answers of the plurality of counterfactual answers being a disfavored counterfactual answer; and
perform fine-tuning on the target model using at least (i) the factual question paired with the preferred factual answer and the disfavored factual answer and (ii) the counterfactual question paired with the preferred counterfactual answer and the disfavored counterfactual answer.

### Claim 16 (depends on 15)

The computer storage medium of claim 15, wherein performing fine-tuning on the target model further includes performing preference-based fine-tuning on the target model using Direct Policy Optimization.

### Claim 17 (depends on 16)

The computer storage medium of claim 16, wherein the instructions further cause the processor to:
identify a first triplet that includes the factual question, a first preferred factual answer, a first disfavored factual answer, and first preference data representing preference for the first preferred factual answer over the first disfavored factual answer;
identify a second triplet that includes the counterfactual question, a first preferred counterfactual answer, a first disfavored counterfactual answer, and second preference data representing preference for the first preferred counterfactual answer over the first disfavored counterfactual answer; and
fine-tune the target model via preference-based fine-tuning using at least the first triplet and the second triplet.

### Claim 18 (depends on 15)

The computer storage medium of claim 15, wherein the counterfactual question is a reformulation of the factual question where a premise and assumption included in the factual question are altered in the counterfactual question such as to contradict the factual question.

### Claim 19 (depends on 15)

The computer storage medium of claim 15, wherein the instructions further cause the processor to:
generate the factual question using a factual question template and inserting a first parameter into the factual question template; and
generate the counterfactual question using a counterfactual question template and inserting said first parameter into the counterfactual question template.

### Claim 20 (depends on 15)

The computer storage medium of claim 15, wherein the instructions further cause the processor to:
submit a cybersecurity query to the target model, the cybersecurity query including a security log from a computing device and a prompt instructing analysis of the security log to identify suspicious activity, causing the target model to generate an output identifying at least one anomalous event from the security log.
