Adversarial training of language models to prevent hijacking of conversational agents
AssigneeCisco Technology, Inc.
InventorsGrégory Mermoud, Pierre-André Savalle, Jean-Philippe Vasseur
Life of this patent
Estimating the term…
Abstract
In one implementation, a device obtains a prompt from a source for input to a language model. The device performs a search for similar prompts to that of the prompt in a prompt database. The device determines, based on results of the search, whether the prompt represents a potential attack. The device prevents a response to the prompt from being returned from the language model to the source, when the device determines that the prompt represents a potential attack.
Claims16
A method comprising:
- obtaining, by a device, a prompt from a source for input to a language model;
- converting, by the device, the prompt into a signature using an attack signature model;
- performing, by the device, a search for similar prompts to that of the prompt in a prompt database by attempting to match the signature of the prompt to those of known attacks in the prompt database;
- determining, by the device and based on results of the search, whether the prompt represents a potential attack;
- and
- preventing, by the device, a response to the prompt from being returned from the language model to the source, when the device determines that the prompt represents a potential attack,
- wherein the prompt database is populated by an adversarial agent that issues prompts to the language model and records whether any of those prompts caused the language model to perform a malicious action.
An apparatus, comprising:
- one or more network interfaces;
- a processor coupled to the one or more network interfaces and configured to execute one or more processes;
- and
- a memory configured to store a process that is executable by the processor, the process when executed configured to:
- obtain a prompt from a source for input to a language model;
- convert the prompt into a signature using an attack signature model;
- perform a search for similar prompts to that of the prompt in a prompt database by attempting to match the signature of the prompt to those of known attacks in the prompt database;
- determine, based on results of the search, whether the prompt represents a potential attack;
- and
- prevent a response to the prompt from being returned from the language model to the source, when the apparatus determines that the prompt represents a potential attack,
- wherein the prompt database is populated by an adversarial agent that issues prompts to the language model and records whether any of those prompts caused the language model to perform a malicious action.
A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:
- obtaining, by the device, a prompt from a source for input to a language model;
- converting, by the device, the prompt into a signature using an attack signature model;
- performing, by the device, a search for similar prompts to that of the prompt in a prompt database by attempting to match the signature of the prompt to those of known attacks in the prompt database;
- determining, by the device and based on results of the search, whether the prompt represents a potential attack;
- and
- preventing, by the device, a response to the prompt from being returned from the language model to the source, when the device determines that the prompt represents a potential attack,
- wherein the prompt database is populated by an adversarial agent that issues prompts to the language model and records whether any of those prompts caused the language model to perform a malicious action.
Dependencies are inferred from the claim text.