Safety alignment for language models based on language model-generated safety categories

A system, comprising:

• at least one memory having executable instructions stored thereon;
• and
• one or more processors configured to execute the executable instructions to cause the system to:
• receive a data set including a plurality of interactions with a language model, each interaction of the plurality of interactions being associated with a predefined safety label included in a taxonomy;
• update the plurality of interactions based on one or more annotation labels provided for the plurality of interactions, wherein at least one of the annotation labels is absent from the taxonomy;
• modify the taxonomy to include the at least one of the annotation labels;
• generate, using an ensemble of generative machine learning models, one or more machine-defined safety labels for each interaction in the plurality of interactions;
• generate a training data set based on revising a label associated with each interaction of the plurality of interactions, the revising being based on a majority vote of the one or more machine-defined safety labels, the predefined safety label associated with each interaction of the plurality of interactions, and the one or more annotation labels provided for the plurality of interactions;
• and
• update the language model based on the training data set, wherein the updating implements guardrails on one or more of a user prompt or an output of the language model such that the language model is restricted from generating responses including unsafe content.

A processor-implemented method for generating a training data set, the method comprising:

• receiving a data set comprising a plurality of interactions, each interaction of the plurality of interactions associated with a predefined safety label;
• for an interaction of the plurality of interactions, generating a plurality of machine-defined safety labels using a corresponding plurality of machine learning models in an ensemble;
• revising the predefined safety label associated with the interaction based on a majority vote of the plurality of machine-defined safety labels, thereby creating a revised interaction;
• and
• generating a training data set for a language model, the training data set comprising the revised interaction.