Retrieval augmented text-to-image generation

A computer-implemented method comprising:

• receiving input text;
• generating, by using a text-to-image model and conditioned on both the input text and image and text pairs selected from a multi-modal knowledge base, an output image, wherein the generating comprises, at each of multiple time steps:
• processing (i) an intermediate representation of the output image for the time step and (ii) the input text using an encoder of the text-to-image model to generate a first feature map for the time step;
• selecting one or more neighbor image and text pairs from the multi-modal knowledge base based on their similarities to the input text;
• for each of the one or more neighbor images and text pairs, processing (i) the image in the neighbor image and text pair and (ii) the text in the neighbor image and text pair using the encoder of the text-to-image model to generate a second feature map for the neighbor image and text pair;
• applying an attention mechanism over the one or more second feature maps using one or more queries derived from the first feature map for the time step to generate an attended feature map;
• and
• generating an updated intermediate representation of the output image for the time step based on using a noise term to de-noise the intermediate representation of the output image, comprising processing the attended feature map for the time step using a decoder of the text-to-image model to generate the noise term.