Generation of curated training data for diffusion models

A method for generating training data, the method comprising steps to:

• match sentences from a sentence dataset with artistic phrases from an artistic phrase dataset to generate a plurality of safe phrases, wherein the plurality of safe phrases are generated by excluding content deemed inappropriate or non-compliant with predetermined content policy guidelines;
• for each of the safe phrases, generate a safe image by, for a predetermined number of iterations, performing steps to:
• input an initial image into a diffusion process to generate a processed image, wherein the diffusion process includes a first diffusion model;
• back-propagate the processed image through a text-image match gradient calculator to calculate a gradient against the safe phrase by generating image embeddings in a shared embedding space based on the processed image, generating text embeddings in the shared embedding space based on the safe phrase, and calculating a differential between the image embeddings and the text embeddings as the gradient;
• and
• update the initial image by iteratively applying the gradient to the processed image during inference;
• and
• pair each of the generated safe images with their respective safe phrase to form a plurality of safe phrase-image pairs, wherein
• the artistic phrases indicate artistic styles in which to render images;
• and
• for each of the plurality of safe phrase-image pairs, each pair comprising a safe image and a respective safe phrase, the respective safe phrase indicates artistic styles in which the safe image was rendered.

A computing system for generating training data, the computing system comprising:

• a sentence dataset;
• an artistic phrase dataset;
• and
• a processor and memory of a computing device, the processor being configured to execute a program using portions of the memory to:
• match sentences from the sentence dataset with artistic phrases from the artistic phrase dataset to generate and output a plurality of safe phrases, wherein the plurality of safe phrases are generated by excluding content deemed inappropriate or non-compliant with predetermined content policy guidelines;
• input an initial image into a diffusion model to generate a processed image;
• calculate a gradient of the processed image against a safe phrase from the plurality of safe phrases by generating image embeddings in a shared embedding space based on the processed image, generating text embeddings in the shared embedding space based on the safe phrase, and calculating a differential between the image embeddings and the text embeddings as the gradient;
• iteratively apply the calculated gradient to the processed image during inference to generate an updated image;
• input the updated image into the diffusion model to repeat the steps of the back-propagation and application of the calculated gradient for a predetermined number of iterations to generate a final safe image;
• pair the safe image with the safe phrase corresponding to the safe image as a safe phrase-image pair;
• and
• train a second diffusion model using the safe phrase-image pair, wherein
• the sentence dataset is curated to exclude inappropriate content according to a predetermined set of guidelines;
• and
• the artistic phrases indicate artistic styles in which to render images, wherein
• the safe phrase corresponding to the safe image indicates artistic styles in which the safe image was rendered.