US 2026/0065082 A1 · HALLUCINATION DETECTION FOR LANGUAGE MODELS

Abstract

Certain embodiments of the disclosure provide techniques for hallucination detection. A method generally includes generating, via a first language model and based on a seed question, a plurality of semantically similar questions; processing the plurality of semantically similar questions with a second language model to generate a plurality of answers; processing the plurality of answers with a third language model to generate a plurality of factual statements; processing the plurality of factual statements with an embedding model to generate a plurality of embeddings; clustering the plurality of embeddings into a plurality of clusters; determining an average proximity score of the plurality of clusters based on a centroid of each of the plurality of clusters; and determining whether the plurality of answers generated by the second language model comprises a hallucination based on a number of the plurality of clusters and the average proximity score of the plurality of clusters.

Claims20

Dependent claims

A method of hallucination detection for language models, comprising:

generating, via a first language model and based on a seed question, a plurality of semantically similar questions;
processing the plurality of semantically similar questions with a second language model to generate a plurality of answers;
processing the plurality of answers with a third language model to generate a plurality of factual statements;
processing the plurality of factual statements with an embedding model to generate a plurality of embeddings;
clustering the plurality of embeddings into a plurality of clusters;
determining an average proximity score of the plurality of clusters based on a centroid of each of the plurality of clusters;
and
determining whether the plurality of answers generated by the second language model comprise a hallucination based on a number of the plurality of clusters and the average proximity score of the plurality of clusters.

5 more elements

A processing system, comprising:

a memory comprising computer-executable instructions;
and
a processor configured to execute the computer-executable instructions and cause the processing system to:
generate, via a first language model and based on a seed question, a plurality of semantically similar questions;
process the plurality of semantically similar questions with a second language model to generate a plurality of answers;
process the plurality of answers with a third language model to generate a plurality of factual statements;
process the plurality of factual statements with an embedding model to generate a plurality of embeddings;
cluster the plurality of embeddings into a plurality of clusters;
determine an average proximity score of the plurality of clusters based on a centroid of each of the plurality of clusters;
and
determine whether the plurality of answers generated by the second language model comprises a hallucination based on a number of the plurality of clusters and the average proximity score of the plurality of clusters.

8 more elements

Dependencies are inferred from the claim text.

Hallucination detection for language models

Life of this patent

Hallucination detection for language models

Life of this patent

Abstract

Claims20