# US 2026/0064980 A1 · Aligning large language models with in-situ user interactions and feedback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064980 A1
- Document ID: US20260064980A1
- Status: Published application
- Kind: A1
- Application number: 19/000,258
- Filed: 2024-12-23
- Published: 2026-03-05
- Earliest claimed: 2024-08-27
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Taiwei SHI; Jennifer Lynay NEVILLE; Sujay Kumar JAUHAR; Mengting WAN; Longqi YANG; Zhuoer WANG; Pei ZHOU
- Web page: https://patentgrep.com/patent/US20260064980A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06N 3/0475: Generative networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A data processing system implements a framework that utilizes in-situ user interactions as a source of feedback for improving the training of LLMs to generate outputs that align with user preferences. The framework includes a user preference evaluation pipeline analyzes in-situ user interactions with the LLMs and generates preference information that can be used to improve the training of the LLM to improve the alignment of the LLM with user preferences. The user preference evaluation pipeline includes a feedback signal identification unit that identifies explicit and/or implicit feedback provided by the users in response to content output by the LLM in response to a user prompt. The feedback signal identification unit estimates user satisfaction with a set of satisfaction rubrics and user dissatisfaction with a set of user dissatisfaction rubrics to generate user preference data that can be used to align an LLM with these user preferences.

## Claims

### Claim 1

A data processing system comprising:
a processor; and
a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
obtaining example data from an example user interaction dataset that includes user interactions between human users and a first large language model, wherein a user interaction includes a user prompt, one or more responses from the first large language model to the user prompt, and one or more user reactions to the one or more responses from the first large language model;
constructing, using a feedback signal identification unit, a first prompt to a second large language model instructing the second large language model to analyze the user interactions between the human users and the first large language model in the example data and to classify the user interactions according to a set of satisfaction rubrics and a set of dissatisfaction rubrics, the set of satisfaction rubrics being indicative of user satisfaction with a response from the first large language model in response to a prompt from a first user, the set of dissatisfaction rubrics being indicative of user dissatisfaction with the response from the first large language model in response to the prompt from the first user;
providing, using the feedback signal identification unit, the first prompt and the example data as an input to the second large language model to cause the second large language model to classify the user interactions between the human users and the first large language model according to the set of satisfaction rubrics and the set of dissatisfaction rubrics and to output feedback signal information;
generating, using a preference data construction unit, preference training data for aligning the first large language model with user preferences expressed in the user interactions with the first large language model; and
performing fine-tuning training of the first large language model using the preference training data to improve alignment of the first large language model with the user preferences.

### Claim 2 (depends on 1)

The data processing system of claim 1, wherein the first large language model is a policy language model for which a policy of the first large language model is to be aligned with the user preferences.

### Claim 3 (depends on 1)

The data processing system of claim 1, wherein the second large language model is a Generative Pre-Trained Transformer (GPT) language model.

### Claim 4 (depends on 1)

The data processing system of claim 1, wherein the example data includes user interactions associated with one or more multi-turn conversational sessions, and wherein to generate the preference training data, the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
constructing, for each multi-turn conversational sessions, a prompt to the second large language model to cause the second large language model to summarize one or more user reactions to responses generated by the first large language model to generate a summary of the user preferences.

### Claim 5 (depends on 4)

The data processing system of claim 4, wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
determining that a respective multi-turn conversational session of the one or more multi-turn conversational sessions was conducted with an expert language model;
constructing, for the respective multi-turn conversational session, a second prompt to the second large language model to cause the second large language model to generate one or more synthetic favored responses based on the summary of the user preferences associated with the respective multi-turn conversational session;
providing the second prompt as an input to the second large language model to obtain the summary of the user preferences; and
constructing, for the respective multi-turn conversational session, a sample to include in the preference training data comprising the user prompt, the one or more synthetic favored responses, and one or more disfavored responses generated by the first large language model.

### Claim 6 (depends on 4)

The data processing system of claim 4, wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
determining that a respective multi-turn conversational session of the one or more multi-turn conversational sessions was conducted with a policy language model;
constructing, for the respective multi-turn conversational session, a second prompt to the first large language model to cause the first large language model to generate one or more synthetic favored responses based on the summary of the user preferences associated with the respective multi-turn conversational session;
providing the second prompt as an input to the first large language model to obtain the summary of the user preferences; and
constructing, for the respective multi-turn conversational session, a sample to include in the preference training data comprising the user prompt, the one or more synthetic favored responses, and one or more disfavored responses generated by the first large language model.

### Claim 7 (depends on 6)

The data processing system of claim 6, wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
constructing the second prompt to include a safety check to prevent the first large language model from generating the summary of the user preferences responsive to the user preferences including one or more unsafe preferences that could cause the first large language model to generate unsafe or offensive content.

### Claim 8 (depends on 1)

The data processing system of claim 1, wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
constructing a series of third prompts for a third language model based on the preference training data;
providing the series of third prompts as an input to the third language model to obtain a series of first responses from the third language model;
constructing a fourth prompt for a fourth language model based on the preference training data;
providing the series of fourth prompts as an input to the fourth language model to obtain a series of second responses from the fourth language model; and
analyzing the series of first response and the series of second responses to determine whether the third language model or the fourth language model is more closely aligned with the user preferences.

### Claim 9

A method implemented in a data processing system for aligning a large language model with user preferences, the method comprising:
obtaining example data from an example user interaction dataset that includes user interactions between human users and a first large language model, wherein a user interaction includes a user prompt, one or more responses from the first large language model to the user prompt, and one or more user reactions to the one or more responses from the first large language model;
constructing, using a feedback signal identification unit, a first prompt to a second large language model instructing the second large language model to analyze the user interactions between the human users and the first large language model in the example data and to classify the user interactions according to a set of satisfaction rubrics and a set of dissatisfaction rubrics, the set of satisfaction rubrics being indicative of user satisfaction with a response from the first large language model in response to a prompt from a first user, the set of dissatisfaction rubrics being indicative of user dissatisfaction with the response from the first large language model in response to the prompt from the first user;
providing, using the feedback signal identification unit, the first prompt and the example data as an input to the second large language model to cause the second large language model to classify the user interactions between the human users and the first large language model according to the set of satisfaction rubrics and the set of dissatisfaction rubrics and to output feedback signal information;
generating, using a preference data construction unit, preference training data for aligning the first large language model with user preferences expressed in the user interactions with the first large language model; and
performing fine-tuning training of the first large language model using the preference training data to improve alignment of the first large language model with the user preferences.

### Claim 10 (depends on 9)

The method of claim 9, wherein the first large language model is a policy language model for which a policy of the first large language model is to be aligned with the user preferences.

### Claim 11 (depends on 9)

The method of claim 9, wherein the second large language model is a Generative Pre-Trained Transformer (GPT) language model.

### Claim 12 (depends on 9)

The method of claim 9, wherein the example data includes user interactions associated with one or more multi-turn conversational sessions, and generating the preference training data further comprises:
constructing, for each multi-turn conversational sessions, a prompt to the second large language model to cause the second large language model to summarize one or more user reactions to responses generated by the first large language model to generate a summary of the user preferences.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
determining that a respective multi-turn conversational session of the one or more multi-turn conversational sessions was conducted with an expert language model;
constructing, for the respective multi-turn conversational session, a second prompt to the second large language model to cause the second large language model to generate one or more synthetic favored responses based on the summary of the user preferences associated with the respective multi-turn conversational session;
providing the second prompt as an input to the second large language model to obtain the summary of the user preferences; and
constructing, for the respective multi-turn conversational session, a sample to include in the preference training data comprising the user prompt, the one or more synthetic favored responses, and one or more disfavored responses generated by the first large language model.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
determining that a respective multi-turn conversational session of the one or more multi-turn conversational sessions was conducted with a policy language model;
constructing, for the respective multi-turn conversational session, a second prompt to the first large language model to cause the first large language model to generate one or more synthetic favored responses based on the summary of the user preferences associated with the respective multi-turn conversational session;
providing the second prompt as an input to the first large language model to obtain the summary of the user preferences; and
constructing, for the respective multi-turn conversational session, a sample to include in the preference training data comprising the user prompt, the one or more synthetic favored responses, and one or more disfavored responses generated by the first large language model.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
constructing the second prompt to include a safety check to prevent the first large language model from generating the summary of the user preferences responsive to the user preferences including one or more unsafe preferences that could cause the first large language model to generate unsafe or offensive content.

### Claim 16

A data processing system comprising:
a processor; and
a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
obtaining example data from an example user interaction dataset that includes user interactions between human users and a policy large language model, wherein a user interaction includes a user prompt, one or more responses from the policy large language model to the user prompt, and one or more user reactions to the one or more responses from the policy large language model;
constructing, using a feedback signal identification unit, a first prompt to an expert large language model instructing the expert large language model to analyze the user interactions between the human users and the policy large language model in the example data and to classify the user interactions according to a set of satisfaction rubrics and a set of dissatisfaction rubrics, the set of satisfaction rubrics being indicative of user satisfaction with a response from the policy large language model in response to a prompt from a first user, the set of dissatisfaction rubrics being indicative of user dissatisfaction with the response from the policy large language model in response to the prompt from the first user;
providing, using the feedback signal identification unit, the first prompt and the example data as an input to the expert large language model to cause the expert large language model to classify the user interactions between the human users and the policy large language model according to the set of satisfaction rubrics and the set of dissatisfaction rubrics and to output feedback signal information;
generating, using a preference data construction unit, preference training data for aligning the policy large language model with user preferences expressed in the user interactions with the policy large language model; and
performing fine-tuning training of the policy large language model using the preference training data to improve alignment of the policy large language model with the user preferences.

### Claim 17 (depends on 16)

The data processing system of claim 16, wherein the policy large language model is a policy language model for which a policy of the policy large language model is to be aligned with the user preferences.

### Claim 18 (depends on 16)

The data processing system of claim 16, wherein the expert large language model is a Generative Pre-Trained Transformer (GPT) language model.

### Claim 19 (depends on 16)

The data processing system of claim 16, wherein the example data includes user interactions associated with one or more multi-turn conversational sessions, and wherein to generate the preference training data, the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
constructing, for each multi-turn conversational sessions, a prompt to the expert large language model to cause the expert large language model to summarize one or more user reactions to responses generated by the policy large language model to generate a summary of the user preferences.

### Claim 20 (depends on 19)

The data processing system of claim 19, wherein the memory further stores executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
determining that a respective multi-turn conversational session of the one or more multi-turn conversational sessions was conducted with an expert language model;
constructing, for the respective multi-turn conversational session, a second prompt to the expert large language model to cause the expert large language model to generate one or more synthetic favored responses based on the summary of the user preferences associated with the respective multi-turn conversational session;
providing the second prompt as an input to the expert large language model to obtain the summary of the user preferences; and
constructing, for the respective multi-turn conversational session, a sample to include in the preference training data comprising the user prompt, the one or more synthetic favored responses, and one or more disfavored responses generated by the policy large language model.
