Generating layered digital designs from sketches using deep learning

A system comprising:

• one or more memory devices;
• and
• one or more processors configured to cause the system to:
• generate, utilizing one or more machine learning models, text embeddings from a compositional reference of a sketch image;
• generate, utilizing an image encoder, image embeddings from the compositional reference of the sketch image;
• and
• generate, utilizing a diffusion model, a digitized digital design of the sketch image by conditioning the diffusion model on the text embeddings and the image embeddings.

A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:

• generating, utilizing a diffusion model, a digitized digital design from a sketch image;
• extracting, utilizing a segmentation model, text regions from the digitized digital design;
• generating, utilizing an inpainting model, a complete background layer by inpainting regions corresponding to the extracted text regions of the digitized digital design;
• and
• generating a digital design document comprising the complete background layer and one or more editable text elements.