System and method for detecting and preventing prompt injection attacks

A network computing device comprising:

• a processor configured with processor-executable instructions to perform operations comprising:
• receiving a prompt from a client computing device, wherein the prompt is configured to invoke a machine-learning application;
• evaluating content of the prompt using one or more pre-filter module;
• determining whether the content of the prompt is safe based on the pre-filter module evaluation and on received context information about a user of the client computing device, wherein the context information includes at least one of the user's identity, location, or role within an organization;
• and
• in response to determining that the content of the prompt is safe:
• inputting the prompt to a main machine-learning model for processing, wherein processing the prompt by the main machine-learning model creates a machine-learning output;
• evaluating content of the machine-learning output using one or more post-filter module;
• and
• determining whether the content of the machine-learning output is safe based on the post-filter module evaluation.

A non-transitory processor readable medium having processor executable instructions stored thereon configured to cause a processor to perform operations comprising:

• receiving a prompt from a client computing device, wherein the prompt is configured to invoke a machine-learning application;
• evaluating content of the prompt using one or more pre-filter module;
• determining whether the content of the prompt is safe based on the pre-filter module evaluation and on received context information about a user of the client computing device, wherein the context information includes one or more of the user's identity, location, or role within an organization;
• and
• in response to determining that the content of the prompt is safe:
• inputting the prompt to a main machine-learning model for processing, wherein processing the prompt by the main machine-learning model creates a machine-learning output;
• evaluating content of the machine-learning output using one or more post-filter module;
• and
• determining whether the content of the machine-learning output is safe based on the post-filter module evaluation.