Training text-to-image model

An electronic device, comprising:

• a processor;
• and
• a memory communicatively connected to the processor, wherein the memory stores instructions executable by the processor, and the instructions, when executed by the processor, cause the processor to perform operations comprising:
• obtaining a first Text-to-Image model and a pre-trained reward model, wherein the first Text-to-Image model is used to generate a corresponding image based on an input text, and the pre-trained reward model is used to score a data pair composed of the input text and the corresponding image;
• and
• adjusting parameters of the first Text-to-Image model based on the pre-trained reward model and a reinforcement learning policy to obtain a second Text-to-Image model, wherein an accumulated reward obtained by the second Text-to-Image model in a generation sequence for implementing Text-to-Image satisfies a preset condition, and the accumulated reward is obtained based on a reward of each stage of the generation sequence;
• wherein the reinforcement learning policy comprises a proximal policy optimization algorithm, wherein the proximal policy optimization algorithm uses a behavior sub-model and an evaluation sub-model, wherein the behavior sub-model is obtained based on initialization of the first Text-to-Image model, and the evaluation sub-model is obtained based on initialization of the pre-trained reward model;
• wherein the generation sequence comprises at least one stage, and wherein the operations further comprise:
• for each stage of the generation sequence:
• generating, by the behavior sub-model, a corresponding output noisy image based on the input text provided;
• and
• outputting, by the evaluation sub-model, the reward of a current stage based on the input text and the output noisy image of the current stage;
• wherein the reward of the current stage comprises a relative entropy between an output of the behavior sub-model in a previous stage prior to the current stage and an output of the behavior sub-model in the current stage;
• and
• wherein the accumulated reward obtained in the generation sequence comprises a total score and a loss item, wherein the total score is obtained by the pre-trained reward model based on an initial input and a final output of the generation sequence, and the loss item is a product of the reward of the last stage of the generation sequence and a loss coefficient.

A non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions, when executed by a processor, are configured to enable a computer to perform operations comprising:

• obtaining a first Text-to-Image model and a pre-trained reward model, wherein the first Text-to-Image model is used to generate a corresponding image based on an input text, and the pre-trained reward model is used to score a data pair composed of the input text and the corresponding image;
• and
• adjusting parameters of the first Text-to-Image model based on the pre-trained reward model and a reinforcement learning policy to obtain a second Text-to-Image model, wherein an accumulated reward obtained by the second Text-to-Image model in a generation sequence for implementing Text-to-Image satisfies a preset condition, and the accumulated reward is obtained based on a reward of each stage of the generation sequence;
• wherein the reinforcement learning policy comprises a proximal policy optimization algorithm, wherein the proximal policy optimization algorithm uses a behavior sub-model and an evaluation sub-model, wherein the behavior sub-model is obtained based on initialization of the first Text-to-Image model, and the evaluation sub-model is obtained based on initialization of the pre-trained reward model;
• wherein the generation sequence comprises at least one stage, and wherein the operations further comprise:
• for each stage of the generation sequence:
• generating, by the behavior sub-model, a corresponding output noisy image based on the input text provided;
• and
• outputting, by the evaluation sub-model, the reward of a current stage based on the input text and the output noisy image of the current stage;
• wherein the reward of the current stage comprises a relative entropy between an output of the behavior sub-model in a previous stage prior to the current stage and an output of the behavior sub-model in the current stage;
• and
• wherein the accumulated reward obtained in the generation sequence comprises a total score and a loss item, wherein the total score is obtained by the pre-trained reward model based on an initial input and a final output of the generation sequence, and the loss item is a product of the reward of the last stage of the generation sequence and a loss coefficient.