# US 2025/0156535 A1 · Determining anomalous tool invocations by applications using large generative models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156535 A1
- Document ID: US20250156535A1
- Status: Published application
- Kind: A1
- Application number: 18/510,331
- Filed: 2023-11-15
- Published: 2025-05-15
- Earliest claimed: 2023-11-15
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Ron KELLER; Idan HEN
- Web page: https://patentgrep.com/patent/US20250156535A1

## CPC

- G06F 21/554: {involving event detection and direct action}
- G06F 21/566: {Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities}
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

This disclosure relates to utilizing a threat detection system to detect anomalous actions provided by a compromised large generative language model (LLM). For instance, the threat detection system utilizes a detection-based large generative model to process select communication between an application system and the LLM and determine when the LLM may have been potentially compromised. In various implementations, utilizing the detection-based large generative model, the threat detection system determines when an LLM is improperly instructing an application system to invoke tools to perform unapproved actions. Furthermore, when an LLM becomes compromised, the threat detection system intelligently safeguards the detection-based large generative model against similar threats that seek to evade detection or compromise the detection-based large generative model.

## Claims

### Claim 1

A computer-implemented method for detecting anomalous application actions, comprising:
identifying a first application request from an application to a large generative language model, the first application request including a system prompt and a user prompt;
obtaining a first output from the large generative language model provided to the application that instructs the application to perform a first action;
in response to the application performing the first action and providing a second application request to the large generative language model, obtaining a second output from the large generative language model instructing the application to perform a second action;
determining that the second output from the large generative language model is anomalous based on using a detection-based large generative model to classify the second output; and
preventing the application from performing the second action based on determining that the second output is anomalous.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the detection-based large generative model does not receive the second application request that the application provides to the large generative language model.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising determining that the first output from the large generative language model is non-anomalous by using the detection-based large generative model to classify the first output based on the system prompt, the user prompt, and the first output.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein using the detection-based large generative model to determine that the second output from the large generative language model is anomalous includes:
providing the first application request including the system prompt and the user prompt, the first output, and the second output to the detection-based large generative model; and
not providing the second application request to the detection-based large generative model.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein:
the application receives external content from an external source in response to performing the first action included in the first output from the large generative language model; and
the application provides the external content to the large generative language model with the second application request.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the external content includes an indirect prompt injection attack to exploit a vulnerability of the large generative language model.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein:
the first application request also includes tools available to the application;
the first action includes utilizing a first tool from the tools to get the external content from the external source; and
the detection-based large generative model determines that the first output is non-anomalous based on the tools and the first action.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein:
the second action includes utilizing a second tool from the tools to access privately stored data; and
the detection-based large generative model determines that the second output is anomalous based on the tools, the first action, and the second action.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, further comprising preventing future access to the external source by the application based on classifying the external source as a threat.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the second application request also includes tools available to the application.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising notifying the application not to perform the second action based on determining that the second output is anomalous.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the application and the detection-based large generative model are both within a same cloud computing system.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, further comprising blocking the application from performing the second action based on determining that the second output is anomalous.

### Claim 14

A computer-implemented method for detecting anomalous application actions, comprising:
identifying a first application request from an application to a large generative language model, the first application request including a system prompt and a user prompt;
obtaining a first output from the large generative language model provided to the application that instructs the application to perform a first action;
in response to the application performing the first action and providing a second application request to the large generative language model, obtaining a second output from the large generative language model instructing the application to perform a second action;
determining that the second output from the large generative language model is anomalous based on using a detection-based machine-learning model to classify the second output; and
notifying the application to not perform the second action based on determining that the second output is anomalous.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein the detection-based machine-learning model is located within a cloud security system that is a separate entity from a cloud computing system that implements the application.

### Claim 16 (depends on 14)

The computer-implemented method of claim 14, wherein the detection-based machine-learning model and the application are implemented within a cloud computing system.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, further comprising preventing the application from performing the second action based on determining that the second output is anomalous.

### Claim 18 (depends on 14)

The computer-implemented method of claim 14, wherein the detection-based machine-learning model is a large generative model trained to detect anomalous actions by the application.

### Claim 19

A system for detecting anomalous application actions, the system comprising:
a processing system; and
a computer memory comprising instructions that, when executed by the processing system, cause the system to perform operations of:
identifying a first application request from an application to a large generative language model, the first application request including a system prompt, a user prompt, and tools available to the application;
obtaining a first output from the large generative language model provided to the application that instructs the application to perform a first action with a first tool from the tools;
in response to the application performing the first action with the first tool and providing a second application request to the large generative language model, obtaining a second output from the large generative language model instructing the application to perform a second action;
determining that the second output from the large generative language model is anomalous based on using a detection-based large generative model to classify the second output; and
preventing the application from performing the second action based on determining that the second output is anomalous.

### Claim 20 (depends on 19)

The system of claim 19, wherein the detection-based large generative model does not receive the second application request that the application provides to the large generative language model.
