System and method for preventing exploitation of large language

A system for preventing exploitation of one or more Large Language Models (LLMs), the system comprising:

• a receiving module to receive one or more Application Programming Interface (API) calls directed to the one or more LLMs, wherein the one or more API calls include at least one of:
• a request submitted by a user to the one or more LLMs, and a response generated by the one or more LLMs for the user;
• an analyzer module to analyze the one or more API calls to detect one or more exploitation risks, including:
• a malicious pattern detector to process the received one or more API calls to generate feature vectors representing contextual meaning and intent of the corresponding API call for flagging suspicious API calls;
• a toxic content detector to analyze the received responses generated by the one or more LLMs and assign a toxicity score to each response to flag a toxic response;
• a plugin vulnerability detector to assess vulnerabilities associated with plugins integrated into the one or more LLMs, including identifying potential malware exploits and remote code execution vulnerabilities;
• and
• a sensitive data detector to analyze the received API calls for detecting private data leaks using a detection model trained using datasets including at least one of:
• Personally Identifiable Information (PII), proprietary code, and proprietary data;
• and an action module to:
• integrate, using a vulnerability detection model, the flagged suspicious API calls, the assigned toxicity score to each response, the assessed vulnerabilities, and the detected private data leaks to generate a comprehensive assessment of the one or more API calls to prioritize detected risks for comprehensive protection of the one or more LLMs;
• perform at least one of:
• blocking the flagged API calls, blocking harmful responses generated by the one or more LLMs, restricting user access to sensitive data, and alerting users to potential data leaks based on the generated comprehensive assessment of the one or more API calls.