Learning apparatus and method for creating image and apparatus and method for image creation

2:

• The learning apparatus for image generation of claim 1, wherein the first generative adversarial network includes:
• a first generator configured to generate a random noise vector from the received sentence feature vector and generate an initial image based on the received sentence feature vector and the generated random noise vector;
• and
• a first discriminator configured to compare the initial image generated by the first generator with a preset first comparison image, determine whether the received image is the first comparison image or the generated initial image according to the comparison result, and feedback the determination result to the first generator.

7:

• A learning method for image generation that is performed in a computing device including one or more processors and a memory storing one or more programs executed by the one or more processors, the learning method comprising:
• at a preprocessing module,
• receiving text for image generation and
• generating a sentence feature vector and a word feature vector from the received text;
• at a first generative adversarial network (GAN),
• receiving the sentence feature vector from the preprocessing module and
• generating an initial image based on the received sentence feature vector;
• and
• at a second generative adversarial network,
• receiving the word feature vector generated by the preprocessing module and the initial image generated by the first generative adversarial network and
• generating a final image based on the word feature vector and the initial image-,
• wherein the generating of the final image includes:
• by a second generator,
• receiving the word feature vector generated by the preprocessing module and the initial image generated by the first generative adversarial network,
• generating an enhanced image from the word feature vector and a feature map of the initial image based on a dynamic memory,
• generating a feature map of the enhanced image from the enhanced image by using a non-local block, and
• generating a final image from the word feature vector and the feature map of the enhanced image based on the dynamic memory;
• and
• by a second discriminator,
• comparing the final image generated by the second generator with a preset second comparison image,
• determining whether the received image is the second comparison image or the generated final image according to the comparison result, and
• feeding back the determination result to the second generator,
• wherein the generating of the final image includes:
• by an image enhancement module,
• generating a key and a value for storage in the dynamic memory by combining the word feature vector with the initial image,
• extracting a key similar to the generated key from among the generated key and a key pre-stored in the dynamic memory to calculate a similarity between the generated key and the extracted key, and
• outputting a weighted sum of the values based on the calculated similarity;
• by an image feature generation module,
• generating an enhanced image based on the output weighted sum and the initial image;
• and
• by a non-local block module,
• generating a feature map of the enhanced image by extracting similar pixels for all regions from the generated enhanced image and resetting the pixels to an average value of similar pixels by using the non-local block.

8:

• The learning method for image generation of claim 7, wherein the generating of the initial image includes:
• by a first generator,
• generating a random noise vector from the received sentence feature vector and
• generating an initial image based on the received sentence feature vector and the generated random noise vector;
• and
• by a first discriminator,
• comparing the initial image generated by the first generator with a preset first comparison image,
• determining whether the received image is the first comparison image or the generated initial image according to the comparison result, and
• feeding back the determination result to the first generator.

13:

• An apparatus for image generation, comprising:
• a preprocessing module configured to receive text for image generation and generate a sentence feature vector and a word feature vector from the received text;
• a first generative adversarial network (GAN) configured to receive the sentence feature vector from the preprocessing module and generate an initial image based on the received sentence feature vector;
• and
• a second generative adversarial network configured to receive the word feature vector generated by the preprocessing module and the initial image generated by the first generative adversarial network and generate a final image based on the word feature vector and the initial image,
• wherein the second generator includes:
• an image enhancement module configured to generate a key and a value for storage in the dynamic memory by combining the word feature vector with the initial image, extract a key similar to the generated key from among the generated key and a key pre-stored in the dynamic memory to calculate a similarity between the generated key and the extracted key, and output a weighted sum of the values based on the calculated similarity;
• an image feature generation module configured to generate an enhanced image based on the output weighted sum and the initial image;
• and
• a non-local block module configured to generate a feature map of the enhanced image by extracting similar pixels for all regions from the generated enhanced image and resetting the pixels to an average value of similar pixels by using the non-local block.

16:

• A method for image generation that is performed in a computing device including:
• one or more processors;
• and
• a memory storing one or more programs executed by the one or more processors, the method comprising:
• at a preprocessing module,
• receiving text for image generation and
• generating a sentence feature vector and a word feature vector from the received text;
• at a first generative adversarial network (GAN),
• receiving the sentence feature vector from the preprocessing module and
• generating an initial image based on the received sentence feature vector;
• and
• at a second generative adversarial network,
• receiving the word feature vector generated by the preprocessing module and the initial image generated by the first generative adversarial network and
• generating a final image based on the word feature vector and the initial image-, wherein the generating of the final image includes:
• by an image enhancement module,
• generating a key and a value for storage in the dynamic memory by combining the word feature vector with the initial image;
• extracting a key similar to the generated key from among the generated key and a key pre-stored in the dynamic memory to calculate a similarity between the generated key and the extracted key;
• and
• outputting a weighted sum of the values based on the calculated similarity;
• by an image feature generation module,
• generating an enhanced image based on the output weighted sum and the initial image;
• and
• by a non-local block module,
• generating a feature map of the enhanced image by extracting similar pixels for all regions from the generated enhanced image and resetting the pixels to an average value of similar pixels by using the non-local block.