# US 2025/0384249 A1 · Generative artificial intelligence model safety

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0384249 A1
- Document ID: US20250384249A1
- Status: Published application
- Kind: A1
- Application number: 18/936,895
- Filed: 2024-11-04
- Published: 2025-12-18
- Earliest claimed: 2024-06-18
- Assignees: Trustwise Inc.
- Applicants: Trustwise Inc.
- Inventors: Manoj SAXENA; Matthew BARKER; Avinash SAXENA; Evan THOMAS; James CARR
- Web page: https://patentgrep.com/patent/US20250384249A1

## CPC

- G06N 3/0475: Generative networks
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method may include providing a query and context associated with the query to a generative artificial intelligence (Gen AI) model, the Gen AI model trained to generate a response to the query based on the context. The method may further include performing analysis of the Gen AI model based on a first relevancy between the query and the context, a second relevancy between the query and the response, and a third relevancy between the response and the context and refining the response based on the analysis.

## Claims

### Claim 1

A method comprising:
providing a query and context associated with the query to a generative artificial intelligence (Gen AI) model, the Gen AI model trained to generate a response to the query based on the context;
performing analysis of the Gen AI model based on a first relevancy between the query and the context, a second relevancy between the query and the response, and a third relevancy between the response and the context; and
refining the response based on the analysis.

### Claim 2 (depends on 1)

The method of claim 1, wherein performing the analysis comprises:
detecting hallucinations in the response; and
identifying causes of the hallucinations based on the first relevancy, the second relevancy, and the third relevancy.

### Claim 3 (depends on 2)

The method of claim 2, wherein the hallucinations include are intrinsic or extrinsic.

### Claim 4 (depends on 2)

The method of claim 2, wherein the first relevancy is analyzed based on context relevancy metric.

### Claim 5 (depends on 2)

The method of claim 2, wherein the second relevancy is analyzed based on answer relevancy metric.

### Claim 6 (depends on 2)

The method of claim 2, wherein the third relevancy is analyzed based on one or more of faithfulness metric or summarization metric.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first relevancy, the second relevancy, and the third relevancy are represented using a first score, a second score, and a third score, respectively.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
obtaining one or more safety policies;
assigning a safety score to the response based on one or more safety policies; and
generating a report including at least the safety score.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
assigning a safety score to the response based on the first relevancy, the second relevancy, and the third relevancy; and
generating a report including at least the safety score.

### Claim 10 (depends on 1)

The method of claim 1, wherein the analysis includes personal identifiable information (PII) detection.

### Claim 11 (depends on 10)

The method of claim 10, wherein the PII detection is performed using a plurality of PII detection models.

### Claim 12

A system comprising:
one or more processors; and
one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the operations comprising:
providing a query and context associated with the query to a generative artificial intelligence (Gen AI) model, the Gen AI model trained to generate a response to the query based on the context;
performing analysis of the Gen AI model based on a first relevancy between the query and the context, a second relevancy between the query and the response, and a third relevancy between the response and the context; and
refining the response based on the analysis.

### Claim 13 (depends on 12)

The system of claim 12, wherein performing the analysis comprises:
detecting hallucinations in the response; and
identifying causes of the hallucinations based on the first relevancy, the second relevancy, and the third relevancy.

### Claim 14 (depends on 13)

The system of claim 13, wherein the hallucinations include are intrinsic or extrinsic.

### Claim 15 (depends on 13)

The system of claim 13, wherein the first relevancy is analyzed based on context relevancy metric.

### Claim 16 (depends on 13)

The system of claim 13, wherein the second relevancy is analyzed based on answer relevancy metric.

### Claim 17 (depends on 13)

The system of claim 13, wherein the third relevancy is analyzed based on one or more of faithfulness metric or summarization metric.

### Claim 18 (depends on 12)

The system of claim 12, wherein the first relevancy, the second relevancy, and the third relevancy are represented using a first score, a second score, and a third score, respectively.

### Claim 19

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a system to perform operations, the operations comprising:
providing a query and context associated with the query to a generative artificial intelligence (Gen AI) model, the Gen AI model trained to generate a response to the query based on the context;
performing analysis of the Gen AI model based on a first relevancy between the query and the context, a second relevancy between the query and the response, and a third relevancy between the response and the context; and
refining the response based on the analysis.

### Claim 20 (depends on 19)

The one or more non-transitory computer-readable media of claim 19, wherein the analysis includes personal identifiable information (PII) detection.
