Detection of cyber attacks driven by compromised large language model applications

A system comprising:

• a processor;
• a data repository in communication with the processor;
• a large language model which, when executed by the processor, generates a first output from a first input comprising at least a prompt injection cyberattack;
• a control application which, when executed by the processor, is programmed to coordinate the first input and the first output of the large language model;
• a controlled application which, when executed by the processor, is programmed to receive, as a second input, the first output from the large language model and to generate a second output using the first output from the large language model;
• and
• a guardian controller comprising:
• a classification machine learning model which, when executed by the processor, is programmed to determine a probability that the first input comprises the prompt injection cyberattack, and
• a security application which, when executed by the processor, enforces a security scheme,
• wherein, when executed by the processor, the guardian controller is programmed to:
• monitor the first output of the large language model;
• determine the probability that the first output of the large language model is poisoned by the prompt injection cyberattack by providing the first output of the large language model to the classification machine learning model, and executing the classification machine learning model to generate the probability;
• determine whether the probability satisfies a threshold;
• enforce, responsive to the probability satisfying the threshold, the security scheme on use of the first output of the large language model by the control application, wherein enforcing the security scheme mitigates the prompt injection cyberattack;
• generate a third output of the controlled application by executing the controlled application using the first output of the large language model;
• and
• return, after enforcing the security scheme, a third output of the controlled application to the control application.

A method of training a classification machine learning model comprising:

• generating, by a control application, a plurality of queries to a large language model, wherein at least some of the plurality of queries comprise known prompt injection cyberattacks;
• generating a plurality of first outputs of the large language model by executing the large language model on at least a subset of the plurality of queries;
• training, iteratively, the classification machine learning model using the plurality of first outputs and a second subset of the plurality of queries until convergence to generate a trained classification machine learning model which, when executed, is trained to detect unknown prompt injection cyberattacks in a plurality of monitored outputs of the large language model;
• receiving, by the trained classification machine learning model, a new output of the large language model;
• determining, by the trained classification machine learning model, that the new output satisfies a threshold;
• and
• enforcing a security scheme on the new output of the large language model prior to transmitting the new output to the control application.