Back to results
US 12,754,908 B2Granted
Trustworthy framework for quality assessment and alignment of generative processes
AssigneeDell Products L.P.
InventorsWerner Spolidoro Freund, Marcus Vinicius Libardi Pazini, Herberth Birck Fröhlich
Life of this patent
Estimating the term…
Abstract
Techniques for testing a RAG system are disclosed. A service obtains source content and facilitates the generation of a question that is designed to test the RAG system. The service facilitates the generation of the question. The service also facilitates the generation of a set of merged excerpts identified from within the source content. Multiple RPIs (reference pattern of information) are generated for the merged excerpts. The service then facilitates the generation of a reference triplet using the question, a specific subset of RPIs, and a specific subset of merged excerpts. The service then subjects the reference triplet to a review process.
Claims20
A method comprising:
- within a retrieval augmented generation (RAG) system, performing an information retrieval operation by:
- (i) accessing a database that is selected to provide source content usable to generate a question designed to test the RAG system and (ii) obtaining the source content;
- receiving, at a question generation module, the source content and a list of previous questions, wherein the question generation module is tasked with generating the question that is designed to test the RAG system;
- receiving, at an excerpt filtering module, the source content and the question generated by the question generation module, wherein the excerpt filtering module is tasked with generating a set of excerpt groups that are identified from within the source content, wherein each excerpt group in the set includes variations of possible excerpts that are fine-tunable during a review process, wherein the each excerpt group comprises excerpt sequences that continuously overlap in textual content such that each excerpt sequence shares at least a portion of text with another excerpt sequence within a same excerpt group, and wherein the excerpt groups are generated by applying a decoding process that enforces extractive constraints on the source content, and wherein the excerpt groups in the set are merged together to form a set of merged excerpts;
- receiving, at a referenced pattern of information (RPI) generation module, the question and the set of merged excerpts, wherein the RPI generation module is tasked with generating a set of RPIs for each merged excerpt included in the set of merged excerpts by constraining generation to content derived from the set of merged excerpts and labeling generated segments as RPIs corresponding to specific excerpts;
- receiving, at a reference processing module, the set of RPIs and the set of merged excerpts, wherein the reference processing module is tasked with selecting, from the set of merged excerpts, a subset of merged excerpts and selecting, from the set of RPIs, a subset of RPIs, wherein the selecting is based on likelihood scores derived from probability outputs of a large language model corresponding to the set of RPIs and the set of merged excerpts, wherein a reference triplet is generated using the subset of merged excerpts and the subset of RPIs, and wherein the reference triplet includes:
- (i) the question, (ii) the subset of RPIs, and (iii) the subset of merged excerpts;
- and
- subjecting the reference triplet to the review process during which the reference triplet is (i) accepted, (ii) discarded, or (iii) curated via fine-tuning, wherein, in response to a modification of the set of merged excerpts during the review process, the RPI generation module is automatically re-invoked to implement partial regeneration by regenerating only RPIs that correspond to those merged excerpts that were modified while preserving at least some previously generated RPIs associated with unmodified merged excerpts.
6 more elements
A computer system comprising:
- one or more processors;
- and
- one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
- within a retrieval augmented generation (RAG) system, perform an information retrieval operation by:
- (i) accessing a database that is selected to provide source content usable to generate a question designed to test the RAG system and (ii) obtaining the source content;
- receive, at a question generation module, the source content and a list of previous questions, wherein the question generation module is tasked with generating the question that is designed to test the RAG system;
- receive, at an excerpt filtering module, the source content and the question generated by the question generation module, wherein the excerpt filtering module is tasked with generating a set of excerpt groups that are identified from within the source content, wherein each excerpt group in the set includes variations of possible excerpts that are fine-tunable during a review process, wherein the each excerpt group comprises excerpt sequences that continuously overlap in textual content such that each excerpt sequence shares at least a portion of text with another excerpt sequence within a same excerpt group, and wherein the excerpt groups are generated by applying a decoding process that enforces extractive constraints on the source content, and wherein the excerpt groups in the set are merged together to form a set of merged excerpts;
- receive, at a referenced pattern of information (RPI) generation module, the question and the set of merged excerpts, wherein the RPI generation module is tasked with generating a set of RPIs for each merged excerpt included in the set of merged excerpts by constraining generation to content derived from the set of merged excerpts and labeling generated segments as RPIs corresponding to specific excerpts;
- receive, at a reference processing module, the set of RPIs and the set of merged excerpts, wherein the reference processing module is tasked with selecting, from the set of merged excerpts, a subset of merged excerpts and selecting, from the set of RPIs, a subset of RPIs, wherein the selecting is based on likelihood scores derived from probability outputs of a large language model corresponding to the set of RPIs and the set of merged excerpts, wherein a reference triplet is generated using the subset of merged excerpts and the subset of RPIs, and wherein the reference triplet includes:
- (i) the question, (ii) the subset of RPIs, and (iii) the subset of merged excerpts;
- and
- subject the reference triplet to the review process during which the reference triplet is (i) accepted, (ii) discarded, or (iii) curated via fine-tuning, wherein, in response to a modification of the set of merged excerpts during the review process, the RPI generation module is automatically re-invoked to implement partial regeneration by regenerating only RPIs that correspond to those merged excerpts that were modified while preserving at least some previously generated RPIs associated with unmodified merged excerpts.
9 more elements
One or more hardware storage devices that store instructions that are executable by one or more processors to cause the one or more processors to:
- within a retrieval augmented generation (RAG) system, perform an information retrieval operation by:
- (i) accessing a database that is selected to provide source content usable to generate a question designed to test the RAG system and (ii) obtaining the source content;
- receive, at a question generation module, the source content and a list of previous questions, wherein the question generation module is tasked with generating the question that is designed to test the RAG system;
- receive, at an excerpt filtering module, the source content and the question generated by the question generation module, wherein the excerpt filtering module is tasked with generating a set of excerpt groups that are identified from within the source content, wherein each excerpt group in the set includes variations of possible excerpts that are fine-tunable during a review process, wherein the each excerpt group comprises excerpt sequences that continuously overlap in textual content such that each excerpt sequence shares at least a portion of text with another excerpt sequence within a same excerpt group, and wherein the excerpt groups are generated by applying a decoding process that enforces extractive constraints on the source content, and wherein the excerpt groups in the set are merged together to form a set of merged excerpts;
- receive, at a referenced pattern of information (RPI) generation module, the question and the set of merged excerpts, wherein the RPI generation module is tasked with generating a set of RPIs for each merged excerpt included in the set of merged excerpts by constraining generation to content derived from the set of merged excerpts and labeling generated segments as RPIs corresponding to specific excerpts;
- receive, at a reference processing module, the set of RPIs and the set of merged excerpts, wherein the reference processing module is tasked with selecting, from the set of merged excerpts, a subset of merged excerpts and selecting, from the set of RPIs, a subset of RPIs, wherein the selecting is based on likelihood scores derived from probability outputs of a large language model corresponding to the set of RPIs and the set of merged excerpts, wherein a reference triplet is generated using the subset of merged excerpts and the subset of RPIs, and wherein the reference triplet includes:
- (i) the question, (ii) the subset of RPIs, and (iii) the subset of merged excerpts;
- and
- subject the reference triplet to the review process during which the reference triplet is (i) accepted, (ii) discarded, or (iii) curated via fine-tuning, wherein, in response to a modification of the set of merged excerpts during the review process, the RPI generation module is automatically re-invoked to implement partial regeneration by regenerating only RPIs that correspond to those merged excerpts that were modified while preserving at least some previously generated RPIs associated with unmodified merged excerpts.
6 more elements
Dependencies are inferred from the claim text.