US 2026/0093735 A1 · COMPUTING CONFIDENCE SCORES FOR LARGE LANGUAGE MODEL-BASED RETRIEVAL AUGMENTATION TOOLS

Abstract

A system for computing confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools includes a host device having a controller. The controller executing programmatic control logic including an algorithm for computing confidence scores for LLM based RAG tools (CLR application). The CLR application receives an input to the LLM from a user and engages an ensemble retriever. The ensemble retriever determines a similarity between the input and predetermined data in the one or more databases. The ensemble retriever generates an output and an output confidence score. The CLR application then causes a human validator to prioritize and review the output in accordance with the output confidence score, where the output is a command to one or more systems of the host device. The system progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on the human validator over time.

Claims20

Dependent claims

A system for computing confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools, the system comprising:

a host device having a controller, the controller having a processor, a memory, and input/output (I/O) ports, the I/O ports in communication with a human-machine interface (HMI) and one or more databases, the processor executing programmatic control logic stored in the memory, the programmatic control logic including an algorithm for computing confidence scores for LLM based RAG tools (CLR application), the CLR application comprising:
a first control logic that receives an input to the LLM from a host device user;
a second control logic that engages an ensemble retriever, wherein the ensemble retriever that determines a similarity between the input and predetermined data in the one or more databases stored in the memory;
a third control logic that causes the ensemble retriever to generate an output and an output confidence score;
and
a fourth control logic that causes a human validator to prioritize and review the output in accordance with the output confidence score, wherein the output is a command to one or more systems of the host device, and wherein the system progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on the human validator over time.

3 more elements

A method for computing confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools, the method comprising:

executing, by a processor of a controller of a vehicle, programmatic control logic stored within memory of the controller, the controller further having input/output (I/O) ports in communication with a human-machine interface (HMI) of the vehicle, the programmatic control logic including an algorithm for computing confidence scores for LLM based RAG tools (CLR application), the CLR application comprising control logic for:
receiving an input to the LLM from a vehicle user via the HMI;
engaging an ensemble retriever, wherein the ensemble retriever that determines a similarity between the input and predetermined data in one or more databases stored in the memory;
causing the ensemble retriever to generate an output and an output confidence score;
and
prioritizing and reviewing the output in accordance with the output confidence score, wherein the output is a command to one or more systems of the vehicle, and wherein the method progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on a human validator over time.

3 more elements

A method for computing confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools, the method comprising:

executing, by a processor of a controller of a vehicle, programmatic control logic stored within memory of the controller, the controller further having input/output (I/O) ports in communication with a human-machine interface (HMI) of the vehicle, the programmatic control logic including an algorithm for computing confidence scores for LLM based RAG tools (CLR application), the CLR application comprising control logic for:
receiving an input to the LLM from a vehicle user via the HMI;
engaging an ensemble retriever, wherein the ensemble retriever that determines a similarity between the input and predetermined data in one or more databases stored in the memory;
causing the ensemble retriever to generate an output and an output confidence score;
prioritizing and reviewing the output in accordance with the output confidence score, wherein the output is a command to one or more systems of the vehicle, and wherein the method progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on a human validator over time;
engaging a semantic tool that extracts semantic information from the input;
and
wherein the semantic tool converts extracted semantic information from the input into a vector in vector space such that the vector defines a mathematical, graphical, vectorized representation of semantic information within the input;
accessing a text vector database stored in memory, wherein the text vector database contains a plurality of predefined semantic text vectors defining mathematical, graphical, vectorized representations of predefined semantic inputs that the vehicle is programmed to accept and respond to;
and
comparing predefined semantic text vectors in the text vector database to vectorized extracted semantic information from the input;
using ranked fusion to calculate a semantic similarity score according to:
a retrieved context Ci1, a vector similarity score, Si, and eliminating context Ci when a similarity score of Ci is such that the vector similarity score Si is less than a threshold T;
fusing “high quality” contexts defined based on an importance score where importance weight is calculated based on a count of context, Ci, in different retrievers, CCi, a rank of a context, Ci, in different retrievers, RCi, and an importance score of a context, Ci, is given as, ICi=f(CCi, RCi);
and
ranking context based on the importance score ICi according to:
Ret
_a
=
[
(
C_a1
,
S_a1
)
,
(
C_a2
,
S_a2
)
,
…
⁡
(
C
an
,
S
an
)
]
Ret
_b
=
[
(
C_b1
,
S_b1
)
,
(
C_b2
,
S_b2
)
,
…
⁡
(
C
bn
,
S
bn
)
]
Ret
_c
=
[
(
C_c1
,
S_c1
)
,
(
C_c2
,
S_c2
)
,
…
⁡
(
C
cn
,
S
cn
)
]
where, context {Cij}, j≥1 is ranked based on similarity score, i.e.:
Si1>Si2 . . . >Sin, wherein “low quality” contexts have vector similarity scores Si less than the threshold T, while “high quality” contexts have importance scores ICi indicating that the input is closely related to or directly implicates critical host device functions;
engaging a syntactic tool that extracts syntactic information from the input;
and
wherein the syntactic tool converts extracted syntactic information from the input into an input raw text vector in vector space such that the vector defines a mathematical, graphical, vectorized representation of syntactic information within the input;
accessing a raw text database stored in memory, wherein the raw text database contains a plurality of predefined raw text vectors defining mathematical, graphical, vectorized representations of predefined raw text inputs that the vehicle is programmed to accept and respond to;
and
comparing predefined raw text vectors in the raw text database to vectorized extracted syntactic information from the input;
calculating a syntactic similarity score according to a Jaccard index and a Levenshtein distance between the vectorized extracted syntactic information from the input and the plurality of predefined raw text vectors in the raw text database;
computing the output confidence score using a normalized weighted sum of the semantic similarity score and the syntactic similarity score according to:
C
out
=
Z
⁡
(
w
i
*
Score
sem
+
w
j
*
Score
syn
)
where Z ( . . . ) is a normalization function, and Wi, Wj are weights, the semantic similarity score is:
Scoresem=f(rank fusion score), and the syntactic similarity score is:
Scoresyn=f(Jaccard Index, Levenshtein Distance, . . . );
assigning the output confidence score to the output, wherein the output commands one or more actuators of the vehicle to adjust performance of relevant vehicle systems;
and
prioritizing and reviewing, by the human validator, the output according to the output confidence score and a ranked context;
and
selectively updating, by the human validator, one or more of the text vector database and the raw text database with new data obtained from the input, the output, and the output confidence score, and
wherein output commands to the one or more actuators of the vehicle alter functionality of the vehicle in accordance with inputs received from the user.

128 more elements

Dependencies are inferred from the claim text.

Computing confidence scores for large language model-based retrieval augmentation tools

Life of this patent

Computing confidence scores for large language model-based retrieval augmentation tools

Life of this patent

Abstract

Claims20