Systems and methods for text-to-image generation using language models

A system for integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model, the system comprising:

• a communication interface that receives a plurality of training samples;
• a memory containing machine readable medium storing machine executable code;
• one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
• receiving, via a data interface, a text prompt describing image content;
• encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space;
• encoding, via the target text encoder, the prompt into a target embedding in a target representation space;
• transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
• generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
• computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
• computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding;
• training the translation network encoder and the translation network decoder based on the first loss and the second loss;
• and
• building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.

A non-transitory machine-readable medium comprising a plurality of machine-executable instructions for integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:

• receiving, via a data interface, a text prompt describing image content;
• encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space;
• encoding, via the target text encoder, the prompt into a target embedding in a target representation space;
• transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
• generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
• computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
• computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding;
• training the translation network encoder and the translation network decoder based on the first loss and the second loss;
• and
• building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.