Method and system for auto multiple image captioning
AssigneeKorea Electronics Technology Institute
InventorsBo Eun Kim, Hye Dong Jung
Life of this patent
Estimating the term…
Abstract
A method and a system for automatically generating multiple captions of an image are provided. A method for training an auto image caption generation model according to an embodiment of the present disclosure includes: generating a caption attention map by using an image; converting the generated caption attention map into a latent variable by projecting the caption attention map onto a latent space; deriving a guide map by using the latent variable; and training to generate captions of an image by using the guide map and the image. Accordingly, a plurality of captions describing various characteristics of an image and including various expressions can be automatically generated.
Claims5
A method for training an auto image caption generation model, the method comprising:
- inputting an image into a convolutional neural network (CNN) and extracting first image characteristics of the inputted image;
- inputting the first image characteristics into a recurrent neural network (RNN);
- generating a first map indicating a portion of the image that influences when a word is generated by using the first image characteristics, and a second map indicating a degree of influence of image information on the portion of the image;
- generating a caption attention map by obtaining a weighted sum of the first map and the second map;
- converting the caption attention map into a latent variable corresponding to a mean and a variance by letting the caption attention map pass through a layer of a latent space;
- sampling the latent variable and deriving a guide map by letting the sampled latent variable pass through the layer;
- combining the image with the guide map;
- inputting the combined image into the CNN and extracting second image characteristics of the combined image;
- calculating a loss between a sentence derived from the extracted characteristics of the combined image and a ground truth sentence;
- and
- generating a caption of the image by using the calculated loss.
A system for automatically generating captions of an image, the system comprising:
- an acquisition unit configured to receive an image;
- and
- a processor configured to
- input an image into a convolutional neural network (CNN) and extract first image characteristics of the inputted image,
- input the first image characteristics into a recurrent neural network (RNN),
- generate a first map indicating a portion of the image that influences when a word is generated by using the first image characteristics, and a second map indicating a degree of influence of image information on the portion of the image,
- generate a caption attention map by obtaining a weighted sum of the first map and the second map,
- convert the caption attention map into a latent variable corresponding to a mean and a variance by letting the caption attention map pass through a layer of a latent space,
- sample the latent variable and derive a guide map by letting the sampled latent variable pass through the layer,
- combine the image with the guide map,
- input the combined image into the CNN and extract second image characteristics of the combined image,
- calculate a loss between a sentence derived from the extracted characteristics of the combined image and a ground truth sentence, and
- generate a caption of the image based on the calculated loss.
Dependencies are inferred from the claim text.