Video and Audio Multimodal Searching System

A computer-implemented method for multimodal searching of video results, the method comprising:

• capturing, by a camera of a user device, video data having a sequence of image frames;
• capturing, by a microphone of the user device, audio data associated with the video data;
• processing, using one or more machine-learned models stored on the user device, the audio data and the sequence of image frames to generate video embeddings, the video embeddings being derived by processing the video data concurrently with the audio data;
• transmitting, to a server, the video embeddings;
• in response to the transmission of the video embeddings and the audio data, receiving one or more video results from the server;
• presenting, on a display of the user device, the one or more video results.

A computing system, the system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
• receiving, by a computing system comprising one or more processors, video data captured by a camera of a user device, the video data having a sequence of image frames;
• receiving audio data associated with the video data captured by the user device;
• processing, using one or more machine-learned models, the sequence of image frames to generate video embeddings related to the sequence of the image frames, the video embeddings having a plurality of image embeddings associated with the sequence of image frames;
• determining one or more video results based on the video embeddings and the audio data;
• and
• transmitting, to the user device, the one or more video results.