# US 12,762,250 B2 · System and method for preventing exploitation of large language > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,250 B2 - Document ID: US12762250B2 - Status: Granted - Kind: B2 - Application number: 19/014,291 - Filed: 2025-01-09 - Published: 2026-07-09 - Granted: 2026-10-13 - Earliest claimed: 2025-01-09 - Assignees: Harness Inc. - Applicants: Harness Inc. - Inventors: Jisheng Wang; Naresh Agarwal; Sanjay Nagraj - Web page: https://patentgrep.com/patent/US12762250B2 ## CPC - G06F 21/577: {Assessing vulnerabilities and evaluating computer system security} - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for preventing exploitation of large language models is disclosed. The method includes receiving and analyzing API calls, including user requests and LLM-generated responses, to detect vulnerabilities. The method uses an encoder-based model to flag suspicious API calls based on contextual analysis, assigns toxicity scores to responses for harmful content, evaluates plugin vulnerabilities for malware risks, and detects private data leaks using trained detection models. Then, a vulnerability detection model integrates flagged API calls, toxicity scores, assessed vulnerabilities, and detected data leaks to generate a comprehensive risk assessment. Based on this assessment, the method performs mitigation actions, including blocking flagged API calls or harmful responses, restricting access to sensitive data, and alerting users to potential leaks. ## Claims ### Claim 1 A system for preventing exploitation of one or more Large Language Models (LLMs), the system comprising: a receiving module to receive one or more Application Programming Interface (API) calls directed to the one or more LLMs, wherein the one or more API calls include at least one of: a request submitted by a user to the one or more LLMs, and a response generated by the one or more LLMs for the user; an analyzer module to analyze the one or more API calls to detect one or more exploitation risks, including: a malicious pattern detector to process the received one or more API calls to generate feature vectors representing contextual meaning and intent of the corresponding API call for flagging suspicious API calls; a toxic content detector to analyze the received responses generated by the one or more LLMs and assign a toxicity score to each response to flag a toxic response; a plugin vulnerability detector to assess vulnerabilities associated with plugins integrated into the one or more LLMs, including identifying potential malware exploits and remote code execution vulnerabilities; and a sensitive data detector to analyze the received API calls for detecting private data leaks using a detection model trained using datasets including at least one of: Personally Identifiable Information (PII), proprietary code, and proprietary data; and an action module to: integrate, using a vulnerability detection model, the flagged suspicious API calls, the assigned toxicity score to each response, the assessed vulnerabilities, and the detected private data leaks to generate a comprehensive assessment of the one or more API calls to prioritize detected risks for comprehensive protection of the one or more LLMs; perform at least one of: blocking the flagged API calls, blocking harmful responses generated by the one or more LLMs, restricting user access to sensitive data, and alerting users to potential data leaks based on the generated comprehensive assessment of the one or more API calls. ### Claim 2 (depends on 1) The system of claim 1, wherein the received one or more API calls are processed using an encoder-only transformer-based model pre-trained using at least prompt injection datasets and flagging of the suspicious API calls further comprises: detecting patterns indicative of malicious and unauthorized activities by comparing the generated feature vectors to known malicious patterns derived from prompt injection datasets; and flagging suspicious API calls based at least on the detected malicious patterns and the contextual meaning. ### Claim 3 (depends on 1) The system of claim 1, wherein flagging the toxicity response further comprises: analyzing the responses generated by the one or more LLMs for harmful and inappropriate content using a model fine-tuned with labeled datasets of toxic and non-toxic language; extracting features including at least one of: keywords, tone, and inflammatory language to classify the analyzed responses into categories, including toxic, harassment, hate speech, and safe; assigning the toxicity score to each response based on the classification; and comparing the toxicity score of each response with a predefined threshold, such that a response is flagged for further review if the toxicity score exceeds the predefined threshold. ### Claim 4 (depends on 1) The system of claim 1, wherein the plugin vulnerability detector further identifies vulnerabilities related to at least one of: insecure input handling, insufficient access control, and remote code execution in the plugins integrated with the one or more LLMs. ### Claim 5 (depends on 1) The system of claim 1, wherein the sensitive data detector uses contextual analysis to identify patterns indicative of sensitive information exposure within the received API calls to detect the private data leaks. ### Claim 6 (depends on 1) The system of claim 1, wherein the plugin vulnerability detector further generates a vulnerability report that includes identified malware exploits and potential mitigation strategies for each assessed plugin. ### Claim 7 (depends on 1) The system of claim 1, wherein the action module blocks flagged API calls based on a predefined threshold of malicious activity. ### Claim 8 (depends on 1) The system of claim 1, wherein the action module restricts user access to sensitive data by redacting portions of the corresponding generated responses identified as private data. ### Claim 9 (depends on 1) The system of claim 1, wherein the action module assigns priority levels to the flagged suspicious API calls, harmful responses, and assessed vulnerabilities to optimize mitigation actions. ### Claim 10 (depends on 1) The system of claim 1, wherein the generated comprehensive assessment is stored in a database for future analysis and iterative improvements in detecting exploitation risks. ### Claim 11 A method for preventing exploitation of one or more Large Language Models (LLMs), the method comprising: receiving one or more Application Programming Interface (API) calls directed to the one or more LLMs, wherein the one or more API calls include at least one of: a request submitted by a user to the one or more LLMs, and a response generated by the one or more LLMs for the user; analyzing the one or more API calls to detect one or more exploitation risks, including: processing the received one or more API calls to generate feature vectors representing contextual meaning and intent of the corresponding API call for flagging suspicious API calls; analyzing the received responses generated by the one or more LLMs and assigning a toxicity score to each response to flag a toxic response; assessing vulnerabilities associated with plugins integrated into the one or more LLMs, including identifying potential malware exploits and remote code execution vulnerabilities; and analyzing the received API calls for detecting private data leaks using a detection model trained using datasets including at least one of: Personally Identifiable Information (PII), proprietary code, and proprietary data; and integrating, using a vulnerability detection model, the flagged suspicious API calls, the assigned toxicity score to each response, the assessed vulnerabilities, and the detected private data leaks to generate a comprehensive assessment of the one or more API calls to prioritize detected risks for comprehensive protection of the one or more LLMs; perform at least one of: blocking the flagged API calls, blocking harmful responses generated by the one or more LLMs, restricting user access to sensitive data, and alerting users to potential data leaks based on the generated comprehensive assessment of the one or more API calls. ### Claim 12 (depends on 11) The method of claim 11, wherein the received one or more API calls are processed using an encoder-only transformer-based model pre-trained using at least prompt injection datasets and flagging of the suspicious API calls further comprises: detecting patterns indicative of malicious and unauthorized activities by comparing the generated feature vectors to known malicious patterns derived from prompt injection datasets; and flagging suspicious API calls based at least on the detected malicious patterns and the contextual meaning. ### Claim 13 (depends on 11) The method of claim 11, wherein flagging the toxicity response further comprises: analyzing the responses generated by the one or more LLMs for harmful and inappropriate content using a model fine-tuned with labeled datasets of toxic and non-toxic language; extracting features including at least one of: keywords, tone, and inflammatory language to classify the analyzed responses into categories, including toxic, harassment, hate speech, and safe; assigning the toxicity score to each response based on the classification; and comparing the toxicity score of each response with a predefined threshold, such that a response is flagged for further review if the toxicity score exceeds the predefined threshold. ### Claim 14 (depends on 11) The method of claim 11, further comprises identifying vulnerabilities related to at least one of: insecure input handling, insufficient access control, and remote code execution in the plugins integrated with the one or more LLMs. ### Claim 15 (depends on 11) The method of claim 11, further comprises using contextual analysis to identify patterns indicative of sensitive information exposure within the received API calls to detect the private data leaks. ### Claim 16 (depends on 11) The method of claim 11, further comprises generating a vulnerability report that includes identified malware exploits and potential mitigation strategies for each assessed plugin. ### Claim 17 (depends on 11) The method of claim 11, further comprises blocking flagged API calls based on a predefined threshold of malicious activity. ### Claim 18 (depends on 11) The method of claim 11, further comprises restricting user access to sensitive data by redacting portions of the corresponding generated responses identified as private data. ### Claim 19 (depends on 11) The method of claim 11, further comprises assigning priority levels to the flagged suspicious API calls, harmful responses, and assessed vulnerabilities to optimize mitigation actions. ### Claim 20 (depends on 11) The method of claim 11, wherein the generated comprehensive assessment is stored in a database for future analysis and iterative improvements in detecting exploitation risks.