Systems and methods for video retrieval and grounding

A system comprising:

• one or more processors configured to execute instructions to cause the system to:
• receive a word-based query for a video;
• encode the word-based query into a query representation using a trained query encoder;
• perform a video retrieval task by identifying, from among a plurality of video representations each representing a respective untrimmed video, one or more similar video representations that are similar to the query representation, each similar video representation representing a respective relevant video;
• perform a video grounding task by generating a grounding for each relevant video by forward propagating each respective similar video representation together with the query representation through a trained grounding module implemented using a trained neural network, each respective similar video representation being used for both the video retrieval task and the video grounding task;
• and
• output one or more identifiers of the one or more relevant videos together with the grounding generated for each relevant video;
• wherein each of the plurality of video representations is generated using a trained video representation generator;
• and
• wherein the video representation generator is trained together with the query encoder using a training dataset of videos having ground-truth multi-sentence annotations, the query encoder being trained to generate a query representation from a given sentence in a given annotation that matches a high-level sentence representation generated by a text-processing branch of the video representation generator from the given sentence and the query representation generated from the given sentence in the given annotation aligns with a high-level clip representation generated by a video-processing branch of the video representation generator from a clip corresponding to the given sentence.

A non-transitory computer readable medium having instructions encoded thereon, wherein the instructions, when executed by one or more processors of a system, cause the system to:

• receive a word-based query for a video;
• encode the word-based query into a query representation using a trained query encoder;
• perform a video retrieval task by identifying, from among a plurality of video representations each representing a respective untrimmed video, one or more similar video representations that are similar to the query representation, each similar video representation representing a respective relevant video;
• perform a video grounding task by generating a grounding for each relevant video by forward propagating each respective similar video representation together with the query representation through a trained grounding module, each respective similar video representation being used for both the video retrieval task and the video grounding task;
• and
• output one or more identifiers of the one or more relevant videos together with the grounding generated for each relevant video;
• wherein each of the plurality of video representations is generated using a trained video representation generator;
• and
• wherein the video representation generator is trained together with the query encoder using a training dataset of videos having ground-truth multi-sentence annotations, the query encoder being trained to generate a query representation from a given sentence in a given annotation that matches a high-level sentence representation generated by a text-processing branch of the video representation generator from the given sentence and the query representation generated from the given sentence in the given annotation aligns with a high-level clip representation generated by a video-processing branch of the video representation generator from a clip corresponding to the given sentence.