System and method of providing search and replace functionality for videos

A method for searching for an element in a video and replacing a portion of the video comprising:

• receiving, from a user, a request to search for specified content in the video, the specified content being one of first input text, a first audio file, a first image file or a first video file;
• presenting, to the user, a user interface (UI) element for loading any of the first input text, the first audio file, the first image file or the first video file to identify the specified content, the UI element also accepting from the user any of a second input text, a second audio file, a second image file or a second video file as replacement content corresponding to the specified content for replacing the specified content in the video;
• generating an index of different elements in the video by frame;
• operating one or more machine-learning (ML) models to identify a matching video clip within the video, the matching video clip being a video clip that includes one or more elements that match the specified content, the one or more elements including one or more of text, audio and video elements, the one or more ML models using the index of the different elements in the video to identify the matching video clip;
• generating a replacement video clip according to a spatio-temporal analysis by the one or more ML models to identify movement of an object or changes in a scene of the specified content;
• generating an audio clip from audio data parsed according to one of silence detection or speaker identification, wherein the audio clip is synchronized to correspond with the replacement video clip according to the spatio-temporal analysis to prevent individual words from being split in the replacement video clip;
• and
• automatically replacing the matching video clip with the replacement video clip and the audio clip to generate a revised video, the revised video comprising the replacement content.

A non-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to perform functions of:

• receiving, from a user, a request to search for specified content in a video, the specified content being one of first input text, a first audio file, a first image file or a first video file;
• presenting, to the user, a user interface (UI) element for loading any of the first input text, the first audio file, the first image file or the first video file to identify the specified content, the UI element also accepting from the user any of a second input text, a second audio file, a second image file or a second video file as replacement content corresponding to the specified content for replacing the specified content in the video;
• generating an index of different elements in the video by frame;
• operating one or more machine-learning (ML) models to identify a matching video clip within the video, the matching video clip being a video clip that includes one or more elements that match the specified content, the one or more elements including one or more of text, audio and video elements, the one or more ML models using the index of the different elements in the video to identify the matching video clip;
• generating a replacement video clip according to a spatio-temporal analysis by the one or more ML models to identify movement of an object or changes in a scene of the specified content;
• generating an audio clip from audio data parsed according to one of silence detection or speaker identification, wherein the audio clip is synchronized to correspond with the replacement video clip according to the spatio-temporal analysis to prevent individual words from being split in the replacement video clip;
• and
• automatically replacing the matching video clip with the replacement video clip and the audio clip to generate a revised video, the revised video comprising the replacement content.