Generating unified embeddings from multi-modal canvas inputs for image retrieval

A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• receiving, from a client device, a multi-modal search input comprising visual input for conducting an image search;
• generating, utilizing a multi-modal embedding neural network, a plurality of region-level embeddings for regions of the multi-modal search input;
• generating, utilizing the multi-modal embedding neural network, a unified embedding from the plurality of region-level embeddings;
• and
• retrieving one or more digital images utilizing the unified embedding that is responsive to the multi-modal search input.

A system comprising:

• at least one memory device comprising a multi-modal embedding neural network;
• and
• at least one processor configured to cause the system to:
• generate a unified embedding for a multi-modal search input comprising visual input by utilizing the multi-modal embedding neural network to:
• determine a plurality of panoptic segments of the multi-modal search input;
• generate a plurality of segment-level semantic embeddings and a plurality of segment-level layout embeddings for the plurality of panoptic segments;
• generate the unified embedding using the plurality of segment-level semantic embeddings and the plurality of segment-level layout embeddings;
• and
• conduct an image search to retrieve one or more digital images that are responsive to the multi-modal search input utilizing the unified embedding.