# US 2025/0165589 A1 · Filtering for harmful generative artificial intelligence results

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165589 A1
- Document ID: US20250165589A1
- Status: Published application
- Kind: A1
- Application number: 18/518,257
- Filed: 2023-11-22
- Published: 2025-05-22
- Earliest claimed: 2023-11-22
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Vinayshekhar BANNIHATTI KUMAR; Sopan KHOSLA; Hayden LAWLER; Siddharth Satish GOYAL; Rashmi GANGADHARAIAH; Brijesh LUCKRIA; James W. HORSLEY; Bhaskar Hosahalli RANGASWAMY; Abhijit S. BARDE
- Web page: https://patentgrep.com/patent/US20250165589A1

## CPC

- G06F 21/55: Detecting local intrusion or implementing counter-measures
- G06F 40/216: using statistical methods
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/44: Statistical methods, e.g. probability models
- G06F 40/56: Natural language generation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques for filtering for harmful generative artificial intelligence (AI) results are described. An example of filtering includes receiving a request for an input phrase to be responded to by a generative AI model; comparing to the received input phrase to at least one known harmful input phrase to determine that the received input phrase is to be provided to the generative AI model; providing the received input phrase to the generative AI model; and generating a response by at least in part on an output of the generative AI model.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving a request for an input phrase to be responded to by a generative artificial intelligence (AI) model;
comparing to the received input phrase to a known harmful input phrase and a plurality of paraphrases associated with the known harmful input phrase in a multi-stage process to determine that the received input phrase is to be provided to the generative AI model;
providing the received input phrase to the generative AI model; and
generating a response by at least in part on an output of the generative AI model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein comparing to the received input phrase to a known harmful input phrase and a plurality of paraphrases associated with the known harmful input phrase in a multi-stage process to determine that the received input phrase is to be provided to the generative AI model comprises:
generating a first set of similarity values between the received input phrase and at least a subset of the stored known harmful input phrases;
determining that the first set of similarity values is below a first similarity threshold;
determining that the first set of similarity values is above a second similarity threshold; and
comparing to the received input phrase to at least one paraphrase associated with a known harmful input phrase that is most similar to the input phrase to determine that the received input phrase is to be passed to the generative AI model by:
generating a second set of similarity values between the received input phrase and the at least one paraphrase associated with a known harmful input phrase that is most similar to the input phrase, and
determining that the second set of similarity values is below a third similarity threshold.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the input phrase is in a format of one of text, audio, or video.

### Claim 4

A computer-implemented method comprising:
receiving a request for an input phrase to be responded to by a generative artificial intelligence (AI) model;
comparing to the received input phrase to at least one known harmful input phrase to determine that the received input phrase is to be provided to the generative AI model;
providing the received input phrase to the generative AI model; and
generating a response by at least in part on an output of the generative AI model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein comparing to the received input phrase to at least one known harmful input phrase to determine that the received input phrase is to be provided to the generative AI model comprises:
generating a first set of similarity values between the received input phrase and at least a subset of the stored known harmful input phrases; and
determining that the first set of similarity values is below a first similarity threshold.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein comparing to the received input phrase to at least one known harmful input phrase to determine that the received input phrase is to be provided to the generative AI model further comprises:
determining that the first set of similarity values is above a second similarity threshold; and
comparing to the received input phrase to at least one paraphrase associated with a known harmful input phrase that is most similar to the input phrase to determine that the received input phrase is to be passed to the generative AI model by:
generating a second set of similarity values between the received input phrase and the at least one paraphrase associated with a known harmful input phrase that is most similar to the input phrase, and
determining that the second set of similarity values is below a third similarity threshold.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein at least one of the first, second, and third similarity thresholds are user configurable.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6, wherein comparing to the received input phrase to at least one known harmful input phrase to determine that the received input phrase is to be provided to the generative AI model comprises:
generating a first set of similarity values between the received input phrase and at least a subset of the stored known harmful input phrases;
determining that none of the first set of similarity values is above a first similarity threshold and is above a second similarity threshold;
generating a second set of similarity values between at least a proper subset stored paraphrases associated with one or more of the stored known harmful input phrases; and
determining that none of the second set of similarity values indicates that the received input phrase is similar to the at least a proper subset stored paraphrases associated with one or more of the stored known harmful input phrases.

### Claim 9 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving one or more known harmful input phrases;
generating one or more paraphrases for each known harmful input phrase using one or more paraphrase generator models; and
storing the known harmful input phrases and the generated one or more paraphrases for each known harmful input phrases.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising:
storing a denial response for one or more of the received harmful input phrases.

### Claim 11 (depends on 4)

The computer-implemented method of claim 4, further comprising:
verifying the output of the generative AI model is not a hallucination.

### Claim 12 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining that the input phrase is not out-of-scope by:
generating an embedding for the input phrase to produce classification logits;
summing the logits in classification logits; and
comparing the sum to an out-of-scope threshold, wherein when the sum is greater than or equal to the out-of-scope threshold the input phrase is not out-of-scope.

### Claim 13 (depends on 4)

The computer-implemented method of claim 4, wherein the input phrase is in a format of one of text, audio, or video.

### Claim 14 (depends on 4)

The computer-implemented method of claim 4, wherein the generative AI model is one of a Transformer-based model, a generative adversarial network, or an auto-regressive convolutional neural networks (AR-CNN).

### Claim 15 (depends on 4)

The computer-implemented method of claim 4, further comprising:
translating the received input phrase to a different language and using the translation as the received input phrase.

### Claim 16

A system comprising:
a first one or more computing devices to implement a storage service in a multi-tenant provider network; and
a second one or more computing devices to implement a generative artificial intelligence (AI) service in the multi-tenant provider network, the generative AI service including instructions that upon execution cause the generative AI service to:
receive a request for an input phrase to be responded to by a generative AI model;
compare to the received input phrase to at least one known harmful input phrase to be stored by the storage service to determine that the received input phrase is to be provided to the generative AI model;
provide the received input phrase to the generative AI model; and
generate a response by at least in part on an output of the generative AI model.

### Claim 17 (depends on 16)

The system of claim 16, wherein the generative AI service is further to compare to the received input phrase to a plurality of paraphrases associated with at least one known harmful input phrase.

### Claim 18 (depends on 16)

The system of claim 16, wherein the generative AI service is further to verify the output of the generative AI model is not a hallucination.

### Claim 19 (depends on 16)

The system of claim 16, wherein the generative AI model is one of a Transformer-based model, a generative adversarial network, or an auto-regressive convolutional neural networks (AR-CNN).

### Claim 20 (depends on 16)

The system of claim 16, wherein the input phrase is in a format of one of text, audio, or video.
