US 2026/0093595 A1 · LANGUAGE MODEL RESPONSE EVALUATION AND ENHANCEMENT

Abstract

The present disclosure generally relates to evaluating and enhancing LLM responses. In some implementations, a system includes multiple language models with different specialized roles that work together to improve response reliability and transparency. A responder model can generate initial responses to user queries, providing diverse perspectives on the same input. An evaluator model can assess and combines responses from the responder models into an accurate and reliable output. A reporter model can generate summaries and alerts about response quality and confidence levels, providing transparency to users about the decision-making process. An artificial intelligence (AI) engine can manage the flow of information between the different models, orchestrating their interactions and ensuring proper sequencing of operations. A retrieval system can provide additional context from external knowledge sources, allowing the system to generate accurate and well-informed responses.

Claims20

Dependent claims

A method comprising:

receiving a user input comprising a prompt and a query;
obtaining contextual information from one or more data sources based on the query;
providing the prompt, the query, and the contextual information to a plurality of responder language models;
receiving a plurality of responses from the plurality of responder language models;
outputting the prompt and the plurality of responses to an evaluator language model that is configured to perform an assessment of the plurality of responses;
receiving the assessment and one or more aggregate responses from the evaluator language model;
providing the prompt and at least one of the assessment or the query to a reporter language model that is configured to generate an alert or summary of the one or more aggregate responses;
receiving the summary or alert from the reporter language model;
and
outputting the one or more aggregate responses and the summary or alert for display on a user interface.

7 more elements

A system comprising:

one or more processors;
and
memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving a user input comprising a prompt and a query;
obtaining contextual information from one or more data sources based on the query;
providing the prompt, the query, and the contextual information to a plurality of responder language models;
receiving a plurality of responses from the plurality of responder language models;
outputting the prompt and the plurality of responses to an evaluator language model that is configured to perform an assessment of the plurality of responses;
receiving the assessment and one or more aggregate responses from the evaluator language model;
providing the prompt and at least one of the assessment or the query to a reporter language model that is configured to generate an alert or summary of the one or more aggregate responses;
receiving the summary or alert from the reporter language model;
and
outputting the one or more aggregate responses and the summary or alert for display on a user interface.

10 more elements

Dependencies are inferred from the claim text.

Language model response evaluation and enhancement

Life of this patent

Language model response evaluation and enhancement

Life of this patent

Abstract

Claims20