Systems and methods for fast novel visual concept learning from sentence descriptions of images

A computer-implemented method for supplemental training of a trained model to describe a novel concept without requiring a complete retraining of the trained model, the method comprising:

• inputting into the trained model a set of images and associated captions related to a novel concept not trained in the trained model, the trained model comprising:
• a language component comprising at least one word embedding layer and a recurrent layer that maps an index of a word in a word dictionary and taken from a caption into a semantic dense word embedding space and stores word context information in the recurrent layer;
• a vision component that generates an image representation of an input image from the set of images;
• a multimodal component, which is distinct from the recurrent layer of the language component, that generates a representation using information from the language component and the vision component;
• and
• a SoftMax layer component that receives input obtained from the multimodal component and predicts an index of a next word;
• fixing at least some of a set of originally learned weights of the trained model during the supplemental training of the trained model;
• and
• fixing at least some of a set of baseline probabilities of the trained model during the supplemental training of the trained model.

A system for providing supplemental training of a trained model to describe a novel concept without requiring a complete retraining of the trained model, the system comprising:

• an input interface for receiving a set of images and associated captions related to a novel concept not trained in the trained model;
• one or more processors;
• and
• a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by the one or more processors, causes steps to be performed comprising:
• inputting into the trained model a set of images and associated captions related to a novel concept not trained in the trained model, the trained model comprising:
• a language component comprising at least one word embedding layer and a recurrent layer that maps an index of a word in a word dictionary and taken from a caption into a semantic dense word embedding space and stores word context information in the recurrent layer;
• a vision component that generates an image representation of an input image from the set of images;
• a multimodal component, which is distinct from the recurrent layer of the language component, that generates a representation using information from the language component and the vision component;
• and
• a SoftMax layer component that receives input obtained from the multimodal component and predicts an index of a next word;
• fixing at least some of a set of originally learned weights of the trained model during the supplemental training of the trained model;
• and
• fixing at least some of a set of baseline probabilities of the trained model during the supplemental training of the trained model.