Edge-based video content search with multimodal content understanding

A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:

• to receive a video signal in an edge computing site of an information processing system configured in accordance with a core-edge architecture;
• to extract key frames from the received video signal;
• for each of at least a subset of the extracted key frames, to generate a multimodal embedding comprising one or more key frame vectors each characterizing one or more of image information, audio information and text information of the extracted key frame;
• and
• to process a search query based at least in part on the key frame vectors;
• wherein processing the search query based at least in part on the key frame vectors comprises comparing an embedding generated for query text of the search query to at least a first key frame vector of a first key frame and one or more additional key frame vectors generated for respective ones of a plurality of adjacent frames of the first key frame.

A method comprising:

• receiving a video signal in an edge computing site of an information processing system configured in accordance with a core-edge architecture;
• extracting key frames from the received video signal;
• for each of at least a subset of the extracted key frames, generating a multimodal embedding comprising one or more key frame vectors each characterizing one or more of image information, audio information and text information of the extracted key frame;
• and
• processing a search query based at least in part on the key frame vectors;
• wherein processing the search query based at least in part on the key frame vectors comprises comparing an embedding generated for query text of the search query to at least a first key frame vector of a first key frame and one or more additional key frame vectors generated for respective ones of a plurality of adjacent frames of the first key frame;
• and
• wherein the method is performed by at least one processing device comprising a processor coupled to a memory.