Action concept enhancement of video-language models in procedural videos

A system for refining a video language model to identify unseen actions, comprising:

• one or more computers and one or more storage devices storing instructions that are executable by the one or more computers to:
• obtain a pretrained video language model that is pretrained on a first set of object and verb pairing datasets to map input video embeddings and action label embeddings, wherein action labels include object and verb pairings, into a shared dimensional space such that cross-modal similarity between an input video and its ground truth action label is maximized;
• generate a random sample of synonyms for each verb in the object and verb pairings of the action labels;
• build digital verb synonym tree structures in which parent nodes in individual verb synonym tree structures include root verbs and child nodes that branch from the parent nodes are the selected synonyms corresponding to the root verb;
• train the pretrained video language model on a second set of object and verb pairing datasets, wherein each object and verb pairing of the second set includes the object in the first set of object and verb pairing datasets and a verb that is a child node of the root verb originally paired with the object in the first set of object and verb pairing datasets, to map input video embeddings and action label embeddings.

A non-transitory computer-readable medium storing software comprising instructions that are executable by one or more computers to refine a video language model to identify unseen actions by:

• obtaining a pretrained video language model that is pretrained on a first set of object and verb pairing datasets to map input video embeddings and action label embeddings, wherein action labels include object and verb pairings, into a shared dimensional space such that cross-modal similarity between an input video and its ground truth action label is maximized;
• generating a random sample of synonyms for each verb in the object and verb pairings of the action labels;
• building digital verb synonym tree structures in which parent nodes in individual verb synonym tree structures include root verbs and child nodes that branch from the parent nodes are the selected synonyms corresponding to the root verb;
• training the pretrained video language model on a second set of object and verb pairing datasets, wherein each object and verb pairing of the second set includes the object in the first set of object and verb pairing datasets and a verb that is a child node of the root verb originally paired with the object in the first set of object and verb pairing datasets, to map input video embeddings and action label embeddings.