Method of processing multimodal tasks, and an apparatus for the same

A method of processing image data, the method comprising:

• obtaining a query from a user input;
• determining a computational resource of an electronic device;
• identifying a sampler neural network corresponding to the computational resource of the electronic device, among a plurality of sampler neural networks;
• obtaining a sequence of frames of one or more input videos;
• selecting frames from the sequence of frames of the one or more input videos, via the sampler neural network configured to extract features from the sequence of frames that are input to the sampler neural network, determine temporal dependencies between the extracted features, and determine an action of selecting or skipping for each of the sequence of frames;
• and
• identifying a video that matches the query via a multimodal neural network configured to receive the selected frames and the query, and output the video that matches the query, among the one or more input videos,
• wherein the sampler neural network and the multimodal neural network are jointly trained based on an aggregated loss that combines an accuracy loss that represents an accuracy of determining the video that matches the query, and an efficiency loss that reflects a proportion of frames being passed to the multimodal neural network, and
• wherein the plurality of sampler neural networks are trained using different weights for combining the accuracy loss and the efficiency loss, the sampler neural network being trained using a weight for a combination of the accuracy loss and the efficiency loss that corresponds to the computational resource of the electronic device.

A non-transitory computer readable storage medium which is configured to, when executed by at least one processor, perform a method of processing image data, the method comprising:

• obtaining a query from a user input;
• determining a computational resource of an electronic device;
• identifying a sampler neural network corresponding to the computational resource of the electronic device, among a plurality of sampler neural networks;
• obtaining a sequence of frames of one or more input videos;
• selecting frames from the sequence of frames of the one or more input videos, via the sampler neural network configured to extract features from the sequence of frames that are input to the sampler neural network, determine temporal dependencies between the extracted features, and determine an action of selecting or skipping for each of the sequence of frames;
• and
• identifying a video that matches the query via a multimodal neural network configured to receive the selected frames and the query, and output the video that matches the query, among the one or more input videos,
• wherein the sampler neural network and the multimodal neural network are jointly trained based on an aggregated loss that combines an accuracy loss that represents an accuracy of determining the video that matches the query, and an efficiency loss that reflects a proportion of frames being passed to the multimodal neural network, and
• wherein the plurality of sampler neural networks are trained using different weights for combining the accuracy loss and the efficiency loss, the sampler neural network being trained using a weight for a combination of the accuracy loss and the efficiency loss that corresponds to the computational resource of the electronic device.