# US 2026/0093744 A1 · Systems and methods for neural network safety moderators

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0093744 A1
- Document ID: US20260093744A1
- Status: Published application
- Kind: A1
- Application number: 19/043,340
- Filed: 2025-01-31
- Published: 2026-04-02
- Earliest claimed: 2024-10-01
- Assignees: Salesforce, Inc.
- Applicants: Salesforce, Inc.
- Inventors: Fan Yin; Xiangyu Peng; Philippe Laban; Yilun Zhou; Chien-Sheng Wu
- Web page: https://patentgrep.com/patent/US20260093744A1

## CPC

- G06F 16/335: Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536)
- G06F 16/353: into predefined classes
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In view of the potential misuse of AI to disseminate dangerous knowledge, embodiments described herein provide an AI safety training framework with a data generation pipeline for generating datasets indicative of multiple different severity levels of potentially harmful LLM-generated text to identify and respond to a potentially harmful query, and/or moderate a potentially harmful LLM-generated text. In this way, an LLM moderator may be integrated as a safety layer within an AI system's architecture, acting as a “firewall” to filter responses generated by another AI chat engine.

## Claims

### Claim 1

A method of training a neural network based language model for moderating safety content, the method comprising:
receiving, via a data interface, a user query and a corresponding neural network generated response;
generating, by one or more neural network based language models, a plurality of revised responses at multiple safety severity levels based on the corresponding neural network generated response and an instruction indicative of the multiple safety severity levels;
training, the one or more neural network based language models, using a first dataset comprising the user query and the plurality of revised responses at multiple safety severity levels to generate a plurality of updated responses at a requested safety severity level;
training, the one or more neural network based language models, using a second dataset comprising the user query, the plurality of updated responses at the requested safety severity level, to predict a severity level associated with a model-generated text; and
building an artificial intelligence (AI) safety moderator agent by deploying the trained one or more neural network based language models on a hardware platform to moderate the model-generated text that is determined to be harmful.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more neural network based language models comprise at least one generator model, and training the at least one generator comprises:
generating, in response to the user query and an instruction describing the multiple safety severity levels, one or more predicted responses at the multiple safety severity levels, respectively; and
updating weights and parameters of the generator model based on a loss comparing the one or more predicted responses and the plurality of revised responses at the multiple safety severity levels, respectively.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
forming the second dataset by augmented responses at the multiple severity levels that are generated by the trained generator model.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more neural network based language models comprise at least one moderator model, and training the at least one moderator model comprises:
generating, in response to the user query, at least one response from the second dataset and an instruction describing the multiple safety severity levels, at least one of:
a first predicted binary classification indicating whether the at least response is safe;
a second predicted binary classification indicating whether the user query is safe; or
a predicted severity level associated with the at least one response; and
updating weights and parameters of the moderator model based on a first loss comparing the first predicted binary classification, the second predicted binary classification, the predicted severity level and ground-truth labels from the second dataset, respectively.

### Claim 5 (depends on 4)

The method of claim 4, wherein the training the at least one moderator model further comprises:
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset.

### Claim 6 (depends on 4)

The method of claim 4, further comprising:
determining the first predicted binary classification is different from a corresponding ground-truth label in the second dataset; and
iteratively re-generating the plurality of revised responses at the multiple safety severity levels from the at least response to form the first dataset in a next training iteration.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
in response to determining, by the AI safety moderator agent, that the model-generated text from another AI agent is harmful:
generating, by the AI safety moderator agent, an updated text at a reduced severity level based on an input of an original query, the model-generated text, and a description of the multiple safety severity levels; and
sending the updated text to the other AI agent in place of the model-generated text.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
in response to determining that an amount of harmful model-generated texts from the other AI agent is greater than a threshold, sending an alert to the other AI agent thereby causing a disablement of the other AI agent through an application programing interface (API).

### Claim 9

A system for training a neural network based language model for moderating safety content, the method comprising, the system comprising:
a memory that stores a plurality of processor executable instructions;
a communication interface that receives a user query and a corresponding neural network generated response; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
generating, by one or more neural network based language models, a plurality of revised responses at multiple safety severity levels based on the corresponding neural network generated response and an instruction indicative of the multiple safety severity levels;
training, the one or more neural network based language models, using a first dataset comprising the user query and the plurality of revised responses at multiple safety severity levels to generate a plurality of updated responses at a requested safety severity level;
training, the one or more neural network based language models, using a second dataset comprising the user query, the plurality of updated responses at the requested safety severity level, to predict a severity level associated with a model-generated text; and
building an artificial intelligence (AI) safety moderator agent by deploying the trained one or more neural network based language models on a hardware platform to moderate the model-generated text that is determined to be harmful.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more neural network based language models comprise at least one generator model, and training the at least one generator comprises:
generating, in response to the user query and an instruction describing the multiple safety severity levels, one or more predicted responses at the multiple safety severity levels, respectively; and
updating weights and parameters of the generator model based on a loss comparing the one or more predicted responses and the plurality of revised responses at the multiple safety severity levels, respectively.

### Claim 11 (depends on 10)

The system of claim 10, wherein the operations further comprise:
forming the second dataset by augmented responses at the multiple severity levels that are generated by the trained generator model.

### Claim 12 (depends on 9)

The system of claim 9, wherein the one or more neural network based language models comprise at least one moderator model, and training the at least one moderator model comprises:
generating, in response to the user query, at least one response from the second dataset and an instruction describing the multiple safety severity levels, at least one of:
a first predicted binary classification indicating whether the at least response is safe;
a second predicted binary classification indicating whether the user query is safe; or
a predicted severity level associated with the at least one response; and
updating weights and parameters of the moderator model based on a first loss comparing the first predicted binary classification, the second predicted binary classification, the predicted severity level and ground-truth labels from the second dataset, respectively.

### Claim 13 (depends on 12)

The system of claim 12, wherein the operation of training the at least one moderator model further comprises:
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset.

### Claim 14 (depends on 13)

The system of claim 13, wherein the operations further comprise:
determining the first predicted binary classification is different from a corresponding ground-truth label in the second dataset; and
iteratively re-generating the plurality of revised responses at the multiple safety severity levels from the at least response to form the first dataset in a next training iteration.

### Claim 15 (depends on 9)

The system of claim 9, wherein the operations further comprise:
in response to determining, by the AI safety moderator agent, that the model-generated text from another AI agent is harmful:
generating, by the AI safety moderator agent, an updated text at a reduced severity level based on an input of an original query, the model-generated text, and a description of the multiple safety severity levels; and
sending the updated text to the other AI agent in place of the model-generated text.

### Claim 16 (depends on 15)

The system of claim 15, wherein the operations further comprise:
in response to determining that an amount of harmful model-generated texts from the other AI agent is greater than a threshold, sending an alert to the other AI agent thereby causing a disablement of the other AI agent through an application programing interface (API).

### Claim 17

A non-transitory machine-readable medium comprising a plurality of machine-executable instructions for training a neural network based language model for moderating safety content, which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, a user query and a corresponding neural network generated response;
generating, by one or more neural network based language models, a plurality of revised responses at multiple safety severity levels based on the corresponding neural network generated response and an instruction indicative of the multiple safety severity levels;
training, the one or more neural network based language models, using a first dataset comprising the user query and the plurality of revised responses at multiple safety severity levels to generate a plurality of updated responses at a requested safety severity level;
training, the one or more neural network based language models, using a second dataset comprising the user query, the plurality of updated responses at the requested safety severity level, to predict a severity level associated with a model-generated text; and
building an artificial intelligence (AI) safety moderator agent by deploying the trained one or more neural network based language models on a hardware platform to moderate the model-generated text that is determined to be harmful.

### Claim 18 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the operations further comprising:
in response to determining, by the AI safety moderator agent, that the model-generated text from another AI agent is harmful:
generating, by the AI safety moderator agent, an updated text at a reduced severity level based on an input of an original query, the model-generated text, and a description of the multiple safety severity levels; and
sending the updated text to the other AI agent in place of the model-generated text.

### Claim 19 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the operations further comprise:
in response to determining that an amount of harmful model-generated texts from the other AI agent is greater than a threshold, sending an alert to the other AI agent thereby causing a disablement of the other AI agent through an application programing interface (API).

### Claim 20 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the one or more neural network based language models comprise at least one moderator model, and training the at least one moderator model comprises:
generating, in response to the user query, at least one response from the second dataset and an instruction describing the multiple safety severity levels, at least one of:
a first predicted binary classification indicating whether the at least response is safe;
a second predicted binary classification indicating whether the user query is safe;
or a predicted severity level associated with the at least one response; and
updating weights and parameters of the moderator model based on a first loss comparing the first predicted binary classification, the second predicted binary classification, the predicted severity level and ground-truth labels from the second dataset, respectively;
generating, in response to the user query, at least one response of a first severity level from the second dataset and an instruction describing the multiple safety severity levels, a moderated version of the at least one response at a second severity level; and
further updating the weights and the parameters of the moderator model based on a second loss comparing the moderated version of the at least one response and a ground-truth response at the second severity level from the second dataset;
determining the first predicted binary classification is different from a corresponding ground-truth label in the second dataset; and
iteratively re-generating the plurality of revised responses at the multiple safety severity levels from the at least response to form the first dataset in a next training iteration.
