Llm techniques for deterministic data analysis and review
AssigneeDryRun Security Co.
InventorsJames Wickett, Kenny Russell Johnson
Life of this patent
Estimating the term…
Abstract
A method of leveraging large language models (LLMs) to produce a deterministic analysis of data is disclosed. The system receives prepared data and a series of prompts that are used to analyze the prepared data. The system iteratively uses the prepared data and a prompt from the series of prompts as input to the LLMs. Each prompt can be designed to result in responses of a specific format or type, thereby reducing random variability in the output of the LLMs after each prompt. The final output of the system is deterministic, depending only on the prepared data and the series of prompts. Furthermore, performing the analysis in steps allows for quickly modifying and optimizing the process.
Claims24
A computer-implemented method for generating a deterministic output from a large language model (LLM), the method comprising:
- receiving an input prompt related to an analysis to be performed on an input data;
- separating the input prompt into a series of assistance questions having incremental complexities, wherein each of the series of assistance questions corresponds to a restricted set of responses expected from an LLM;
- iteratively performing:
- providing an assistance question of the series of assistance questions to the LLM;
- analyzing a response from the LLM;
- and
- determining, based on the response, whether assistance data is required to generate an adequate response to the assistance question;
- and
- in response to determining that the assistance data is required, supplying the assistance data to the LLM and prompting the LLM to generate an updated response to the assistance question using the assistance data;
- updating the series of assistance questions based on the response or the updated response;
- and
- determining a final response based on responses to the series of assistance questions including the updated response.
A system comprising:
- at least one processor;
- and
- at least one memory storing instructions that, when executed by the at least one processor, cause the system to:
- receive instructions to perform an analysis on an input data;
- compose, based on the instructions, a series of assistance questions, wherein each of the series of assistance questions corresponds to a restricted set of responses expected from a large language model (LLM);
- iteratively perform steps comprising:
- provide, to one or more LLMs, an assistance question of the series of assistance questions;
- analyze a response from the one or more LLMs;
- determine, based on the response, whether assistance data is required to generate an adequate response to the assistance question;
- in response to determining that the assistance data is required, supply the assistance data to the one or more LLMs and prompt the one or more LLMs to generate an updated response to the assistance question using the assistance data;
- and
- update the series of assistance questions based on the response or the updated response;
- and
- determine a final response based on the responses to the series of assistance questions including the updated response.
At least one non-transitory, computer-readable storage medium comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a system, cause the system to:
- receive an input prompt;
- compose a series of incremental prompts based on the input prompt,
- wherein each of the series of incremental prompts corresponds to a response format expected from a large language model (LLM);
- iteratively perform steps comprising:
- provide, to one or more LLMs, an incremental prompt from the series of incremental prompts;
- analyze a response from the one or more LLMs;
- determine, based on the response, whether assistance data is required to generate an adequate response to the incremental prompt;
- in response to determining that the assistance data is required, supply the assistance data to the one or more LLMs and prompt the one or more LLMs to generate an updated response to the incremental prompt using the assistance data;
- and
- update the series of incremental prompts based on the response or the updated response;
- and
- determine a final response based on responses to the series of incremental prompts including the updated response.
A computer-implemented method, comprising:
- receiving a set of code segments and a code context,
- wherein the set of code segments is generated based in part on the code context and a particular aspect of security concerns;
- providing, to one or more Large Language Models (LLMs), one or more prompts corresponding to the particular aspect of security concerns;
- receiving a set of initial responses from the one or more LLMs;
- determining, based on the set of initial responses, that assistance data is required for analysis of the set of code segments;
- supplying, based on a set of initial responses from the one or more LLMs, assistance data for the set of code segments,
- wherein the assistance data is supplied to the one or more LLMs during analysis of the set of code segments and before a determination of whether a security vulnerability exists;
- receiving a set of updated responses from the one or more LLMs after supplying the assistance data;
- determining whether a security vulnerability exists based on a set of updated responses from the one or more LLMs;
- and
- generating a security analysis report detailing the determination,
- wherein results of the determination comprise at least an indication of whether a security vulnerability pertaining to the particular aspect of security concerns has been detected.
Dependencies are inferred from the claim text.