# US 12,118,471 B2 · Mitigation for prompt injection in A.I. models capable of accepting text input > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,471 B2 - Document ID: US12118471B2 - Status: Granted - Kind: B2 - Application number: 18/143,512 - Filed: 2023-05-04 - Published: 2023-11-09 - Granted: 2024-10-15 - Earliest claimed: 2022-05-04 - Assignees: Preamble, Inc - Applicants: Preamble, Inc. - Inventors: Jonathan Cefalu; Jeremy Charles McHugh; Ron Heichman - Web page: https://patentgrep.com/patent/US12118471B2 ## CPC - G06N 3/092: Reinforcement learning - G06F 40/279: Recognition of textual entities - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0475: Generative networks - G06N 3/091: Active learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system for use with an artificial intelligence (AI) model configured to accept text input, such as generative pre-trained transformer (GPT), that detects and tags trusted instructions and nontrusted instructions of an input provided by a user responsive to an AI model prompt. The system uses reinforcement learning (RL) and a set of rules to remove the untrusted instructions from the input and provide only trusted instructions to the AI model. The input is represented as tokens, wherein the trusted instructions and the untrusted instructions are represented using incompatible token sets. ## Claims ### Claim 1 A system, comprising: an artificial intelligence (AI) model configured to accept electronic text input comprising instructions including a sequence of tokens in response to an AI model prompt and configured to use deep learning to produce human-like text responsive to the instructions; and a processor configured to: receive the instructions including the sequence of tokens; apply reinforcement learning (RL) during operation of the AI model to determine electronic trusted instructions and electronic untrusted instructions from the electronic text input provided responsive to the AI model prompt; electronically tag the electronic trusted instructions with a trusted tag and electronically tag the electronic untrusted instructions with an untrusted tag, the trusted tag and the untrusted tag being represented by token vectors respectively indicating a trusted or untrusted source of the sequence of tokens, wherein the trusted tag and untrusted tag are respectively indicative of a type of instruction including at least a trusted instruction represented by a first token set adapted to be executed by the AI model to produce trusted human-like text responsive to the trusted instruction and an untrusted instruction represented by a second token set incompatible with the first token set and adapted to not be executed by the AI model; and apply RL to modify the instructions including the sequence of tokens provided in response to the AI model prompt to indicate that instructions tagged with the trusted tag and represented by the first token set are to be obeyed and that instructions tagged with the untrusted tag and represented by the second token set are to be disregarded and to remove instructions tagged with the untrusted tag from the sequence of tokens provided responsive to the AI model prompt to create instructions to the AI model that are influenced by the instructions tagged with the trusted tag but not influenced by the instructions tagged with the untrusted tag, wherein the AI model is configured to execute the instructions in the electronic text input that has been tagged with the trusted tag to provide the trusted human-like text. ### Claim 2 (depends on 1) The system of claim 1, wherein the RL is reinforcement learning from human feedback (RLHF). ### Claim 3 (depends on 1) The system of claim 1, wherein the type of instruction further comprises a semi-trusted instruction, and wherein semi-trusted instructions are disregarded in the modified instructions by the processor. ### Claim 4 (depends on 1) The system of claim 1, wherein the incompatible token sets have separate incompatible dictionaries. ### Claim 5 (depends on 1) The system of claim 1, wherein applying RL by the processor to modify the instructions comprises the RL automatically deleting instructions tagged with the untrusted tag from the electronic text input before being entered into the AI model. ### Claim 6 (depends on 1) The system of claim 1, wherein applying RL by the processor to determine electronic untrusted instructions comprises applying a set of rules to the sequence of tokens provided responsive to the AI model prompt. ### Claim 7 (depends on 6) The system of claim 6, wherein the rules are configured to be custom configured by a user. ### Claim 8 (depends on 1) The system of claim 1, wherein electronically tagging the electronic trusted instructions with a trusted tag and electronically tagging the electronic untrusted instructions with an untrusted tag comprises the processor electronically tagging each token of the sequence of tokens of the instructions with either a trusted tag or an untrusted tag. ### Claim 9 (depends on 8) The system of claim 8, wherein electronically tagging the electronic trusted instructions with a trusted tag and electronically tagging the electronic untrusted instructions with an untrusted tag comprises the processor using the tags to keep track of which tokens of the sequence of tokens come from a user and which tokens of the sequence of tokens come from a trusted application prompt. ### Claim 10 (depends on 1) The system of claim 1, wherein the AI model is trained to follow an instruction of a trusted sequence of tokens and the AI model is penalized for following any instruction received in full or in part from an untrusted sequence of tokens. ### Claim 11 (depends on 1) The system of claim 1, wherein the processor is further configured to: detect a non-conforming hidden command to the AI model in the electronic text input; and modify the electronic text input to remove the non-conforming hidden command. ### Claim 12 (depends on 1) The system of claim 1, wherein the AI model is a generative pretrained transformer (GPT), and wherein the processor is trained to modify operation of the GPT. ### Claim 13 (depends on 1) The system of claim 1, wherein the processor is configured to remove the instructions tagged with the untrusted tag from the electronic text input in a way that is hidden from a user entering the electronic text input. ### Claim 14 (depends on 1) The system of claim 1, wherein the processor is configured to identify users entering instructions tagged with the untrusted tag in a report configured to allow management to understand and address users entering potential violating commands in the electronic text input. ### Claim 15 (depends on 14) The system of claim 14, wherein the processor generates the report in real-time. ### Claim 16 (depends on 1) The system of claim 1, wherein instructions tagged with an untrusted tag include words having attributes directed to at least one of cyberbullying, harassment, toxicity, islamophobia, misogyny, or journalistic qualities. ### Claim 17 A method of instructing an artificial intelligence (AI) model configured to accept electronic text input comprising instructions including a sequence of tokens in response to an AI model prompt and to perform deep learning to produce human-like text responsive to the instructions, the method comprising: applying, by a processor, reinforcement learning (RL) during operation of the AI model to the instructions to determine electronic trusted instructions and electronic untrusted instructions from the electronic text input provided responsive to the AI model prompt; electronically tagging, by the processor, the electronic trusted instructions with a trusted tag and electronically tagging the electronic untrusted instructions with an untrusted tag, the trusted tag and the untrusted tag being represented by token vectors respectively indicating a trusted or untrusted source of the sequence of tokens, wherein the trusted tag and untrusted tag are respectively indicative of a type of instruction including at least a trusted instruction represented by a first token set adapted to be executed by the AI model to produce trusted human-like text responsive to the trusted instruction and an untrusted instruction represented by a second token set incompatible with the first token set and adapted to not be executed by the AI model; applying, by the processor, RL to modify the instructions including the sequence of tokens provided in response to the AI model prompt to indicate that instructions tagged with the trusted tag and represented by the first token set are to be obeyed and that instructions tagged with the untrusted tag and represented by the second token set are to be disregarded and to remove instructions tagged with the untrusted tag from the sequence of tokens provided responsive to the AI model prompt to create instructions to the AI model that are influenced by the instructions tagged with the trusted tag but not influenced by the instructions tagged with the untrusted tag; and executing, by the AI model, the instructions in the electronic text input that has been tagged with the trusted tag to provide the trusted human-like text. ### Claim 18 (depends on 17) The method of claim 17, further comprising dynamically updating a tag of an instruction based on real-time feedback or monitoring of suspicious behavior from a user.