Performing video moment retrieval utilizing deep learning
AssigneeAdobe Inc.
InventorSeunghyun Yoon
Life of this patent
Estimating the term…
Abstract
The present disclosure relates to systems, methods, and non-transitory computer-readable media that learns parameters for a natural language video localization model utilizing a curated dataset. In particular, in some embodiments, the disclosed systems generate a set of similarity scores between a target query and a video dataset that includes a plurality of digital videos. For instance, the disclosed systems determines a false-negative threshold by utilizing the set of similarity scores to exclude a subset of false-negative samples from the plurality of digital videos. Further, the disclosed systems determines a negative sample distribution and generates a curated dataset that includes a subset of negative samples with the subset of false-negative samples excluded.
Claims20
A computer-implemented method comprising:
- generating, utilizing a text embedding model, a set of similarity scores between a target query and a video dataset comprising a plurality of digital videos;
- determining a false-negative threshold by utilizing the set of similarity scores to exclude a subset of false-negative samples from the plurality of digital videos;
- determining a negative sample distribution for the plurality of digital videos based on the target query with the subset of false-negative samples excluded;
- generating a curated dataset comprising a subset of negative samples based on the negative sample distribution without the subset of false-negative samples;
- and
- learning parameters for a natural language video localization model utilizing the curated dataset.
A system comprising:
- one or more memory devices comprising a pre-trained natural language video localization model, wherein the pre-trained natural language video localization model is pre-trained on a curated dataset based on a negative sample distribution with a subset of false-negative samples removed;
- and
- one or more processors configured to cause the system to:
- process a search query from a client device that indicates one or more concepts utilizing the pre-trained natural language video localization model by:
- generating a text embedding from the search query;
- determining a set of digital video embeddings from a dataset of digital videos;
- and
- comparing the text embedding from the search query with the set of digital video embeddings from the dataset of digital videos;
- and
- provide one or more indications of video content from one or more videos responsive to the search query based on the comparison between the set of digital video embeddings and the text embedding.
A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
- generating, utilizing a text embedding model, a set of similarity scores between a target query and a video dataset comprising a plurality of digital videos;
- determining a false-negative threshold for the plurality of digital videos;
- identifying a subset of false-negative samples of the plurality of digital videos to exclude based on the set of similarity scores and the false-negative threshold;
- determining a negative sample distribution for the plurality of digital videos based on the target query with the subset of false-negative samples excluded;
- identifying a subset of negative samples of the plurality of digital videos based on the negative sample distribution;
- generating a curated dataset comprising the identified subset of negative samples without the subset of false-negative samples;
- and
- learning parameters of a natural language video localization model based on the curated dataset.
Dependencies are inferred from the claim text.