Security marker injection for large language models
AssigneeIntuit Inc.
InventorsItsik Yizhak Mantin, Ron Bitton, Yael Mathov Gome, Gal Cohen
Life of this patent
Estimating the term…
Abstract
A method includes receiving, at a server from a user device, a user query to a large language model (LLM), creating an LLM query from the user query, inserting an security marker instruction into the LLM query to trigger an injection of a security marker, and sending the LLM query to the LLM. The method further includes receiving, from the LLM, an LLM response to the LLM query, evaluating the LLM response to detect whether the security marker is present in the LLM response, and setting a prompt injection signal based on whether the security marker is present in the LLM response.
Claims14
A method comprising:
- receiving, at a server from a first user device, a first user query to a large language model (LLM);
- creating a first LLM query from the first user query;
- inserting a first security marker instruction into the first LLM query to trigger an injection of a security marker, wherein the security marker comprises individually swapping at least one predefined character in a first LLM response with at least one different predefined character;
- sending the first LLM query to the LLM;
- receiving, from the LLM, the first LLM response to the first LLM query;
- evaluating the first LLM response to detect that the security marker is absent from the first LLM response based on the at least one predefined character being in the first LLM response instead of being swapped for the at least one different character;
- setting a first prompt injection signal as malicious based on determining that the security marker is absent from the first LLM response;
- and
- stopping transmission of any user response having any portion of the first LLM response based on the first prompt injection signal being set as malicious.
A system comprising:
- at least one computer processor;
- a large language model (LLM) query manager executing on the at least one computer processor and configured to:
- receive, from a first user device, a first user query to an LLM,
- create a first LLM query from the first user query,
- send the first LLM query to the LLM, and
- receive, from the LLM, a first LLM response to the first LLM query;
- and
- an LLM firewall executing on the at least one computer processor and configured to:
- insert a first security marker instruction into the first LLM query to trigger an injection of a security marker, wherein the security marker comprises individually swapping at least one predefined character in the first LLM response with at least one different predefined character,
- evaluate the first LLM response to detect that the security marker is absent from the first LLM response based on the at least one predefined character being in the first LLM response instead of being swapped for the at least one different character,
- set a first prompt injection signal as malicious based on determining that the security marker is absent from the first LLM response, and
- stop transmission of any user response having any portion of the first LLM response based on the first prompt injection signal being set as malicious.
Dependencies are inferred from the claim text.