Computer-implemented methods for machine learning model based spatial-temporal adaptive shift for end-to-end text-video retrieval

A computer-implemented method comprising:

• generating a set of embeddings for each of a plurality of sections of each frame of a plurality of frames of a video;
• determining, by one or more machine learning models, a proper subset of the plurality of sections of a frame of the plurality of frames for a time shift based on the set of embeddings;
• time shifting the proper subset of the plurality of sections of the frame to generate a time shifted frame;
• generating an updated set of embeddings based on the time shifted frame;
• receiving a search request comprising input text;
• determining the video is a match for the search request based on the input text and the updated set of embeddings for the time shifted frame;
• and
• generating an indication based on the match.

A non-transitory computer-readable medium storing code that, when executed by a device, causes the device to perform a method comprising:

• generating a set of embeddings for each of a plurality of sections of each frame of a plurality of frames of a video;
• determining, by one or more machine learning models, a proper subset of the plurality of sections of a frame of the plurality of frames for a time shift based on the set of embeddings;
• time shifting the proper subset of the plurality of sections of the frame to generate a time shifted frame;
• generating an updated set of embeddings based on the time shifted frame;
• receiving a search request comprising input text;
• determining the video is a match for the search request based on the input text and the updated set of embeddings for the time shifted frame;
• and
• generating an indication based on the match.