Conditional image synthesis

A non-transitory computer readable medium storing code for image processing, the code comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• obtaining a prompt input and a shape input, wherein the prompt input describes an image element and the shape input indicates a shape for the image element;
• generating a conditioned noise map performing a noise map optimization using a shape loss based on the shape input;
• and
• generating, using an image generation model, a synthetic image based on the prompt input and the conditioned noise map, wherein the synthetic image depicts the image element with the shape indicated by the shape input.

A system comprising:

• a memory component;
• and
• a processing device coupled to the memory component;
• a condition encoder comprising parameters stored in the memory component and configured to encode a condition input representing a color palette to obtain a condition embedding;
• and
• an image generation model configured to generate a conditioned noise map by performing a noise map optimization using the condition embedding and to generate a synthetic image based on a prompt input and the conditioned noise map.