System, method, and computer program for multimodal video retrieval
AssigneeSamsung Electronics Co., Ltd.
InventorsZhiming Hu, Ning Ye, Iqbal Ismail Mohomed, Salar Hosseini Khorasgani
Life of this patent
Estimating the term…
Abstract
Provided are system, method, and device for performing multimodal video retrieval. According to embodiments, the method may include: obtaining a first plurality of frames of a video; selecting a second plurality of frames from among the first plurality of frames using a frame selection module, wherein a number of the second plurality of frames may be less than a number of the first plurality of frames; determining a representation of the video based on the selected second plurality of frames using a neural network model; and storing the representation of the video in a memory.
Claims20
A method comprising:
- obtaining a first plurality of frames of a video;
- selecting a second plurality of frames from among the first plurality of frames using a frame selection module based on a plurality of first features extracted from the first plurality of frames, wherein a number of the second plurality of frames is less than a number of the first plurality of frames;
- extracting a plurality of second features from the second plurality of frames;
- generating a plurality of weights corresponding to the second plurality of frames using a neural network model;
- generating a representation of the video by calculating a weighted mean based on the plurality of weights and the plurality of second features;
- and
- storing the representation of the video in a memory.
A system comprising:
- a memory storage storing computer-executable instructions;
- and
- at least one processor communicatively coupled to the memory storage, wherein the at least one processor is configured to execute the instructions to:
- obtain a first plurality of frames of a video;
- select a second plurality of frames from among the first plurality of frames using a frame selection module based on a plurality of first features extracted from the first plurality of frames, wherein a number of the second plurality of frames is less than a number of the first plurality of frames;
- extract a plurality of second features from the second plurality of frames;
- generate a plurality of weights corresponding to the selected second plurality of frames using a neural network model;
- generate a representation of the video by calculating a weighted mean based on the plurality of weights and the plurality of second features;
- and
- store the representation of the video in the memory.
A non-transitory computer-readable recording medium having recorded thereon instructions executable by at least one processor to cause the at least one processor to perform a method comprising:
- obtaining a first plurality of frames of a video;
- selecting a second plurality of frames from among the first plurality of frames using a frame selection module, wherein a number of the second plurality of frames is less than a number of the first plurality of frames;
- generating a plurality of weights corresponding to the selected second plurality of frames using a neural network model;
- generating a representation of the video by calculating a weighted mean based on the plurality of weights and the plurality of second features;
- and
- storing the representation of the video in a memory.
Dependencies are inferred from the claim text.