# US 2025/0384145 A1 · Large Language Model Response to Jailbreaks with Self-Correction and Correction with External Feedback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0384145 A1
- Document ID: US20250384145A1
- Status: Published application
- Kind: A1
- Application number: 19/236,322
- Filed: 2025-06-12
- Published: 2025-12-18
- Earliest claimed: 2024-06-13
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Swetasudha Panda; Michael Louis Wick; Jason Anthony Peck; Naveen Jafer Nizar
- Web page: https://patentgrep.com/patent/US20250384145A1

## CPC

- G06F 21/577: {Assessing vulnerabilities and evaluating computer system security}
- G06F 2221/033: Test or assess software
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and apparatus to implement techniques to correct for jailbreak prompts input to generative language models are described. A prompt is generated that includes a jailbreak prompt, an original response to the jailbreak prompt, and a correction response. The generated response is then submitted to a generative language model and response returned.

## Claims

### Claim 1

A system, comprising:
at least one processor;
a memory, comprising program instructions that when executed by the at least one processor cause the at least one processor to implement a machine learning model development system, the machine learning model development system configured to:
for a generative language model under test:
identify a jailbreak defense technique that comprises generating a prompt to an evaluation model for the generative language model under test, wherein the prompt includes:
a jailbreak prompt;
an original language model response; and
a corrective prompt; and
create a jailbreak test pipeline that includes the generative language model under test and the jailbreak defense technique;
execute the jailbreak test pipeline using a plurality of different test inputs including at least one jailbreak attempt;
capture one or more performance metrics of the jailbreak technique during execution of the jailbreak test pipeline, wherein the one or more performance metrics comprise a least one over-refusal metric of the generative language model under test; and
providing, via an interface of the machine learning model development system, the one or more performance metrics of the jailbreak technique.

### Claim 2 (depends on 1)

The system of claim 1, wherein the original language model response is generated by the generative language model.

### Claim 3 (depends on 1)

The system of claim 1, wherein the original language model response is generated by a different generative language model.

### Claim 4 (depends on 1)

The system of claim 1, wherein the prompt further includes one or more examples of response refinement.

### Claim 5 (depends on 1)

The system of claim 1, wherein the over-refusal metric is at least one of an over-refusal accuracy, an over-refusal precision, an over-refusal recall, or an over-refusal F1 score.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more performance metrics further comprise at least one attack success metric.

### Claim 7 (depends on 1)

The system of claim 1, wherein the generative language model under test and the jailbreak defense technique are selected according to one or more requests received via the interface of the machine learning model development system.

### Claim 8

A computer-implemented method, comprising:
for a generative language model under test:
identifying a jailbreak defense technique that comprises generating a prompt to an evaluation model for the generative language model under test, wherein the prompt includes:
a jailbreak prompt;
an original language model response; and
a corrective prompt; and
creating a jailbreak test pipeline that includes the generative language model under test and the jailbreak defense technique;
executing the jailbreak test pipeline using a plurality of different test inputs including at least one jailbreak attempt;
capturing one or more performance metrics of the jailbreak technique during execution of the jailbreak test pipeline, wherein the one or more performance metrics comprise at least one over-refusal metric of the generative language model under test; and
providing, via an interface of machine learning model development system, the one or more performance metrics of the jailbreak technique.

### Claim 9 (depends on 8)

The method of claim 8, wherein the original language model response is generated by the generative language model.

### Claim 10 (depends on 8)

The method of claim 8, wherein the original language model response is generated by a different generative language model.

### Claim 11 (depends on 8)

The method of claim 8, wherein the prompt further includes one or more examples of response refinement.

### Claim 12 (depends on 8)

The method of claim 8, wherein the over-refusal metric is at least one of an over-refusal accuracy, an over-refusal precision, an over-refusal recall, or an over-refusal F1 score.

### Claim 13 (depends on 8)

The method of claim 8, wherein the one or more performance metrics further comprise at least one attack success metric.

### Claim 14 (depends on 8)

The method of claim 8, wherein the generative language model under test and the jailbreak defense technique are selected according to one or more requests received via the interface of the machine learning model development system.

### Claim 15

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices, cause the one or more computing devices to implement:
for a generative language model under test:
identifying a jailbreak defense technique that comprises generating a prompt to an evaluation model for the generative language model under test, wherein the prompt includes:
a jailbreak prompt;
an original language model response; and
a corrective prompt; and
creating a jailbreak test pipeline that includes the generative language model under test and the jailbreak defense technique;
executing the jailbreak test pipeline using a plurality of different test inputs including at least one jailbreak attempt;
capturing one or more performance metrics of the jailbreak technique during execution of the jailbreak test pipeline, wherein the one or more performance metrics comprise at least one over-refusal metric of the generative language model under test; and
providing, via an interface of machine learning model development system, the one or more performance metrics of the jailbreak technique.

### Claim 16 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, wherein the original language model response is generated by the generative language model.

### Claim 17 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, wherein the original language model response is generated by a different generative language model.

### Claim 18 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, wherein the prompt further includes one or more examples of response refinement.

### Claim 19 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, wherein the over-refusal metric is at least one of an over-refusal accuracy, an over-refusal precision, an over-refusal recall, or an over-refusal F1 score.

### Claim 20 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, wherein the one or more performance metrics further comprise at least one attack success metrics.
