Auto-extract system with keyword, ranking, and prompt generation

A method for generating an extractor for a target data entity comprising:

• prompting one or more language models to extract the target data entity from each batch of a plurality of batches of chunks using extraction instructions in an extraction prompt, each batch of the plurality of batches including chunks generated from content of a submission of a plurality of submissions and one or more chunks having a ground truth value for the target data entity in the submission;
• dividing the plurality of batches into a first set of batches for which the ground truth value was extracted and a second set of batches failing to extract the ground truth value;
• prompting the one or more language models with a first metaprompt using each failing batch in the second set of batches, the first metaprompt comprising the failing batch, and a request to (i) identify root cause causing the failing batch to fail extraction and (ii) identify additional extraction instructions to improve the extraction for the failing batch;
• prompting the one or more language models with one or more second metaprompts comprising the extraction instructions, the root cause for each failing batch in the second set of batches, the additional extraction instructions for each failing batch, and a request to integrate the additional extraction instructions into the extraction instructions;
• repeating dividing the plurality of batches into the first set of batches for which the ground truth value was extracted and the second set of batches failing to extract the ground truth value using the integrated extraction instructions, prompting the one or more language models with the first metaprompt, and prompting the one or more language models with the one or more second metaprompts until at least one of:
• the first set of batches satisfy an extraction accuracy criterion;
• or
• a maximum iterations criterion is satisfied;
• and
• providing the integrated extraction instructions in the extraction prompt to extract a value for the target data entity from another submission.

A system for automatic generation of an extractor for a target data entity, the system comprising:

• one or more processing circuits configured to:
• receive prompt training data comprising a plurality of submissions, each submission comprising a plurality of chunks related to submission documents, one or more chunks having a ground truth value for the target data entity and a subset of the plurality of chunks not having the ground truth value;
• prompt one or more language models with the ground truth value for each submission and a request to identify the one or more chunks having the ground truth value associated with the submission;
• prompt the one or more language models with a first keyword metaprompt comprising the one or more chunks having the ground truth value from each submission and a request to identify keywords correlated to the target data entity;
• divide the one or more chunks having the ground truth value into identified chunks and unidentified chunks, wherein the identified chunks are identified by performing a keyword search across the plurality of chunks for each submission of the plurality of submissions using the keywords;
• prompt the one or more language models with a second keyword metaprompt comprising the unidentified chunks, the keywords, and a request to identify additional keywords correlated to the ground truth values in the unidentified chunks;
• append the additional keywords to the keywords;
• repeat the first keyword metaprompt or the second keyword metaprompt using the appended keywords and append the additional keywords until at least one the identified chunks satisfy of a coverage criterion or a first maximum iterations criterion is satisfied;
• prompt the one or more language models with a request to rank a plurality of first batches of chunks according to ranking instructions of a ranking prompt, each first batch of the plurality of first batches comprising one or more chunks from a submission;
• divide the plurality of first batches of chunks into a first set of first batches passing ranking and a second set of first batches failing the ranking, wherein the first set of first batches passing the ranking are identified by (i) executing the ranking prompt upon each first batch of the plurality of first batches and (ii) responsive to the one or more chunks having the ground truth value in the first batch satisfying an order criterion, adding to the first set of batches passing the ranking the first batch;
• prompt the one or more language models with a ranking analysis metaprompt using each failed batch in the second set of first batches failing the ranking, the ranking analysis metaprompt comprising the failed batch, the ranking instructions from the ranking prompt, and a request to (i) identify a root cause that caused the failed batch to fail the ranking and (ii) identify additional ranking instructions to improve the ranking for the failed batch;
• prompt the one or more language models with one or more ranking optimization metaprompts comprising the ranking instructions, the root cause for each failed batch, the additional ranking instructions for each failed batch, a request to summarize the root causes from each failed batch, a request to group the root causes from each failed batch according to a reason for failure, a request to provide a count of the root causes within each group, and a request to integrate the additional ranking instructions into the ranking instructions;
• repeat dividing the plurality of first batches of chunks into the first set of first batches passing the ranking and the second set of first batches failing the ranking, prompting the one or more language models with the ranking analysis metaprompt and prompting the one or more language models with the one or more ranking optimization metaprompts until at least one of the first set of first batches satisfy a rank accuracy criterion or a second maximum iterations criterion is satisfied, wherein the first set of first batches passing the ranking are identified using the ranking prompt after the additional ranking instructions have been integrated;
• prompt the one or more language models with a seed extraction metaprompt comprising:
• one or more pairs, each pair comprising (i) a second batch of chunks and (ii) the ground truth value corresponding to the second batch, at least one chunk of the second batch having the ground truth value corresponding to the second batch;
• and
• a request to generate an extraction prompt to extract the ground truth value for the target data entity from the second batch of chunks for the pair;
• prompt the one or more language models to extract a value from a third batch of chunks, each third batch including chunks from one submission, using the extraction prompt;
• divide a plurality of third batches of chunks into a first set of third batches for which the ground truth value was extracted and a second set of third batches failing to extract the ground truth value;
• prompt the one or more language models with a extraction analysis metaprompt using each batch failing extraction in the second set of third batches failing to extract the ground truth value, the extraction analysis metaprompt comprising the batch failing extraction, extraction instructions from the extraction prompt, and a request to (i) identify an extraction root cause that caused the batch failing extraction to fail extraction and (ii) identify additional extraction instructions to improve the extraction for the batch failing extraction;
• prompt the one or more language models with one or more extraction optimization metaprompts comprising the extraction instructions, the extraction root cause for each batch failing extraction, the additional extraction instructions for each batch failing extraction, a request to summarize the extraction root cause from each batch failing extraction, a request to group the extraction root causes from each batch failing extraction according to a reason for failure, a request to provide a count of the extraction root causes within each group of the extraction root causes, and a request to integrate the additional extraction instructions into the extraction instructions;
• repeat dividing the plurality of third batches of chunks into the first set of third batches for which the ground truth value was extracted and the second set of third batches failing to extract the ground truth value, prompting the one or more language models with the extraction analysis metaprompt, and prompting the one or more language models with the extraction optimization metaprompt until at least one of the first set of third batches satisfy an extraction accuracy criterion or a third maximum iterations criterion is satisfied, where in the first set of third batches for which the ground truth value was extracted are identified using the extraction prompt after the additional extraction instructions have been integrated;
• and
• extract a value for the target data entity from another submission using the keywords, the ranking prompt and the extraction prompt.