# US 2024/0403560 A1 · Prevention of prompt injection attacks on large language models by tokenization of structured data elements

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0403560 A1
- Document ID: US20240403560A1
- Status: Published application
- Kind: A1
- Application number: 18/446,314
- Filed: 2023-08-08
- Published: 2024-12-05
- Earliest claimed: 2023-06-02
- Assignees: CrowdStrike, Inc.
- Applicants: CrowdStrike, Inc.
- Inventors: Daniel Radu; Marian Radu; Sven Krasser
- Web page: https://patentgrep.com/patent/US20240403560A1

## CPC

- G06F 21/6227: {where protection concerns the structure of data, e.g. records, types, queries}
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 16/908: using metadata automatically derived from the content
- G06F 21/604: {Tools and structures for managing or administering access control systems}
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for implementing prevention of prompt injection attacks on large language models by tokenization of structured data elements is presented. The systems and methods replace one or more data elements in a database response with one or more tokens to produce a tokenized database response. The systems and methods provide the tokenized database response to a large language model (LLM). The systems and methods receive a tokenized LLM output that includes at least one of the one or more tokens. The systems and methods produce a detokenized LLM output by replacing the one or more tokens in the tokenized LLM output with the one or more data elements.

## Claims

### Claim 1

A method comprising:
replacing one or more data elements in a database response with one or more tokens to produce a tokenized database response;
providing the tokenized database response to a large language model (LLM);
receiving a tokenized LLM output that comprises at least one of the one or more tokens; and
producing, by a processing device, a detokenized LLM output by replacing the at least one of the one or more tokens in the tokenized LLM output with at least one of the one or more data elements.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more data elements are a plurality of data elements, and wherein the database response corresponds to a query of a database, the method further comprising:
identifying a first portion of the plurality of data elements that are modifiable by an external entity; and
replacing the first portion of the plurality of data elements with the one or more tokens.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
identifying a second portion of the plurality of data elements that are not modifiable by the external entity; and
maintaining the second portion of the plurality of data elements in the tokenized database response.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
extracting metadata from the database response; and
analyzing the metadata to determine which of the plurality of data elements are modifiable by the external entity.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
evaluating the one or more data elements in the database response to generate heuristics data corresponding to the one or more data elements;
evaluating the heuristics data to determine which of the one or more data elements comprise malicious content; and
replacing the one or more data elements comprising the malicious content with the one or more tokens.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
encoding one or more subsequent data elements in a subsequent database response to produce an encoded database response comprising one or more encoded data elements;
providing the encoded database response to the LLM, wherein the LLM produces an encoded LLM output based on the encoded database response, wherein the encoded LLM output comprises at least one of the one or more encoded data elements; and
decoding the at least one of the one or more encoded data elements in the encoded LLM output to produce a decoded LLM output.

### Claim 7 (depends on 1)

The method of claim 1, wherein the tokenized LLM output comprises a tokenized API call, the method further comprising:
producing a detokenized API call by replacing the at least one of the one or more tokens in the tokenized API call with at least one of the one or more data elements;
executing the detokenized API call to produce the database response; and
providing the detokenized LLM output to a user interface.

### Claim 8

A system comprising:
a processing device; and
a memory to store instructions that, when executed by the processing device cause the processing device to:
replace one or more data elements in a database response with one or more tokens to produce a tokenized database response;
provide the tokenized database response to a large language model (LLM);
receive a tokenized LLM output that comprises at least one of the one or more tokens; and
produce a detokenized LLM output by replacing the at least one of the one or more tokens in the tokenized LLM output with at least one of the one or more data elements.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more data elements are a plurality of data elements, the database response corresponds to a query of a database, and wherein the processing device, responsive to executing the instructions, further causes the system to:
identify a first portion of the plurality of data elements that are modifiable by an external entity;
identifying a second portion of the plurality of data elements that are not modifiable by the external entity;
replace the first portion of the plurality of data elements with the one or more tokens; and
maintain the second portion of the plurality of data elements in the tokenized database response.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processing device, responsive to executing the instructions, further causes the system to:
extract metadata from the database response; and
analyze the metadata to determine which of the plurality of data elements are modifiable by the external entity.

### Claim 11 (depends on 8)

The system of claim 8, wherein the processing device, responsive to executing the instructions, further causes the system to:
evaluate the one or more data elements in the database response to generate heuristics data corresponding to the one or more data elements;
evaluate the heuristics data to determine which of the one or more data elements comprise malicious content; and
replace the one or more data elements comprising the malicious content with the one or more tokens.

### Claim 12 (depends on 8)

The system of claim 8, wherein the processing device, responsive to executing the instructions, further causes the system to:
encode one or more subsequent data elements in a subsequent database response to produce an encoded database response comprising one or more encoded data elements;
provide the encoded database response to the LLM, wherein the LLM produces an encoded LLM output based on the encoded database response, wherein the encoded LLM output comprises at least one of the one or more encoded data elements; and
decode the at least one of the one or more encoded data elements in the encoded LLM output to produce a decoded LLM output.

### Claim 13 (depends on 8)

The system of claim 8, wherein the LLM is a first LLM, and wherein the processing device, responsive to executing the instructions, further causes the system to:
provide the tokenized LLM output and an instruction to a second LLM; and
perform, by the second LLM, the instruction based on the tokenized LLM output.

### Claim 14 (depends on 8)

The system of claim 8, wherein the tokenized LLM output comprises a tokenized API call, and wherein the processing device, responsive to executing the instructions, further causes the system to:
produce a detokenized API call by replacing the at least one of the one or more tokens in the tokenized API call with at least one of the one or more data elements;
execute the detokenized API call to produce the database response; and
provide the detokenized LLM output to a user interface.

### Claim 15

A non-transitory computer readable medium, having instructions stored thereon which, when executed by a processing device, cause the processing device to:
replace one or more data elements in a database response with one or more tokens to produce a tokenized database response;
provide the tokenized database response to a large language model (LLM);
receive a tokenized LLM output that comprises at least one of the one or more tokens; and
produce, by the processing device, a detokenized LLM output by replacing the at least one of the one or more tokens in the tokenized LLM output with at least one of the one or more data elements.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the one or more data elements are a plurality of data elements, the database response corresponds to a query of a database, and wherein the processing device is to:
identify a first portion of the plurality of data elements that are modifiable by an external entity;
identify a second portion of the plurality of data elements that are not modifiable by the external entity;
replace the first portion of the plurality of data elements with the one or more tokens; and
maintain the second portion of the plurality of data elements in the tokenized database response.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the processing device is to:
extract metadata from the database response; and
analyze the metadata to determine which of the plurality of data elements are modifiable by the external entity.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the processing device is to:
evaluate the one or more data elements in the database response to generate heuristics data corresponding to the one or more data elements;
evaluate the heuristics data to determine which of the one or more data elements comprise malicious content; and
replace the one or more data elements comprising the malicious content with the one or more tokens.

### Claim 19 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the processing device is to:
encode one or more subsequent data elements in a subsequent database response to produce an encoded database response comprising one or more encoded data elements;
provide the encoded database response to the LLM, wherein the LLM produces an encoded LLM output based on the encoded database response, wherein the encoded LLM output comprises at least one of the one or more encoded data elements; and
decode the at least one of the one or more encoded data elements in the encoded LLM output to produce a decoded LLM output.

### Claim 20 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the tokenized LLM output comprises a tokenized API call, and wherein the processing device is to:
produce a detokenized API call by replacing the at least one of the one or more tokens in the tokenized API call with at least one of the one or more data elements;
execute the detokenized API call to produce the database response; and
provide the detokenized LLM output to a user interface.
