Generating natural language descriptions of images
AssigneeGoogle Inc.
InventorsSamy Bengio, Oriol Vinyals, Alexander Toshkov Toshev, Dumitru Erhan
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating descriptions of input images. One of the methods includes obtaining an input image; processing the input image using a first neural network to generate an alternative representation for the input image; and processing the alternative representation for the input image using a second neural network to generate a sequence of a plurality of words in a target natural language that describes the input image.
Claims20
A method performed by one or more computers, the method comprising:
- obtaining an input image;
- processing the input image using a first neural network to generate an alternative representation for the input image;
- and
- processing the alternative representation for the input image using a second neural network to generate a sequence of words in a target natural language that describes the input image, including using the second neural network to select words for inclusion in the sequence of words until a special stop word is selected that identifies an end to the sequence of words, wherein each word in the sequence of words after an initial word is selected by conditioning the second neural network on a preceding word in the sequence of words.
A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- obtaining an input image;
- processing the input image using a first neural network to generate an alternative representation for the input image;
- and
- processing the alternative representation for the input image using a second neural network to generate a sequence of words in a target natural language that describes the input image, including using the second neural network to select words for inclusion in the sequence of words until a special stop word is selected that identifies an end to the sequence of words, wherein each word in the sequence of words after an initial word is selected by conditioning the second neural network on a preceding word in the sequence of words.
A computer program product encoded on one or more non-transitory computer storage media, the computer program product comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
- obtaining an input image;
- processing the input image using a first neural network to generate an alternative representation for the input image;
- and
- processing the alternative representation for the input image using a second neural network to generate a sequence of words in a target natural language that describes the input image, including using the second neural network to select words for inclusion in the sequence of words until a special stop word is selected that identifies an end to the sequence of words, wherein each word in the sequence of words after an initial word is selected by conditioning the second neural network on a preceding word in the sequence of words.
Dependencies are inferred from the claim text.