Recurrent neural network architectures which provide text describing images

A system for providing an output phrase describing an image, comprising:

• means for creating feature maps describing image features in location in the image, wherein the feature maps are created by processing the image with a convolutional neural network (CNN) trained to extract the image features based on color values of pixels within the locations;
• means for processing the image features of the image using a first long short-term memory (LSTM) neural network to obtain a first object word and a second object word characterizing the image features, wherein processing the image features includes processing the feature maps with the first LSTM neural network;
• means for processing the image features, the first object word, and the second object word with a second LSTM neural network to obtain a first attribute word and a second attribute word;
• and
• means for providing an output phrase describing the image, the output phrase including the first attribute word modifying the first object word, and the second attribute word modifying the second object word.

A non-transitory computer-readable medium storing instructions, which, when executed by at least one computing device, cause the computing device to:

• process image features of an image using a first long short-term memory (LSTM) neural network to obtain a first object word and a second object word characterizing the image features;
• process the image features, the first object word, and the second object word with a second LSTM neural network to obtain a first attribute word and a second attribute word;
• and
• provide an output phrase describing the image, the output phrase including the first attribute word modifying the first object word, and the second attribute word modifying the second object word, wherein providing the output phrase comprises merging the first object word and the second object word from the first LSTM neural network, and the first attribute word and the second attribute word from the second LSTM neural network, to obtain the output phrase.