Modeling semantic concepts in an embedding space as distributions

A system to annotate images with determined text labels to describe content of the image, the system comprising:

• one or more processors;
• and
• computer-readable storage media having stored thereon instructions that are executable by the one or more processors to perform operations comprising:
• processing a training image having multiple text labels, said processing including generating a set of image regions that correspond to respective labels of the multiple text labels;
• embedding the set of image regions within an embedding space representing semantic concepts as distributions rather than representing the semantic concepts as single points, the semantic concepts represented being described by text labels of a text vocabulary and capable of being depicted in image content, the set of image regions embedded with distributions representing the semantic concepts depicted in the image content of the set of image regions, the distributions in the embedding space positioned based on semantic relationships determined for the text labels, and determination of the semantic relationships being based on meanings of the text labels of the text vocabulary;
• determining a set of semantically meaningful image regions of a query image, the set of semantically meaningful image regions of the query image being mappable to the text labels in the embedding space;
• processing the set of semantically meaningful image regions of the query image to discard the semantically meaningful image regions of the query image that fail to meet at least one predefined criterion and to obtain a subset of semantically meaningful image regions of the query image that meet the at least one predefined criterion;
• determining the text labels that describe depicted semantic concepts of the query image by mapping criteria-meeting image regions of the query image to the distributions of the embedding space;
• and
• annotating the query image with at least two of the determined text labels.

One or more computer-readable storage media comprising instructions stored thereon that, responsive to execution by a computing device, perform operations comprising:

• generating an embedding space representing both images and text labels of a text vocabulary, including:
• computing distributions representing semantic concepts in the embedding space rather than representing the semantic concepts as single points, the semantic concepts for which the distributions are computed being described by respective text labels of the text vocabulary and capable of being depicted in image content;
• determining semantic relationships between meanings of the text labels of the text vocabulary;
• positioning the distributions in the embedding space based on the semantic relationships determined for the respective text labels;
• and
• mapping representative images to the distributions of the embedding space, wherein the image content depicted by the representative images exemplifies corresponding semantic concepts of the distributions;
• determining a set of semantically meaningful image regions of a query image, the set of semantically meaningful image regions of the query image being mappable to the text labels in the embedding space;
• processing the set of semantically meaningful image regions of the query image to discard semantically meaningful image regions of the query image that fail to meet at least one predefined criterion and to obtain a subset of the semantically meaningful image regions of the query image that meet the at least one predefined criterion;
• determining, using the embedding space, at least one of the text labels of the embedding space describing at least one depicted semantic concept in criteria-meeting image regions of the query image;
• and
• annotating the query image by associating the determined text labels with the query image.