Active Sparse Labeling of Video Frames

A system for active sparse labeling of multimedia input to facilitate dense video understanding, the system comprising at least one computer processor configured to:

• a. receive a multimedia input comprising a plurality of frames;
• b. apply an adaptive proximity-aware uncertainty selection model to iteratively select a subset of frames from the multimedia input, the model configured to:
• i. calculate frame-level uncertainty for each frame based on pixel-wise confidence scores of localization;
• ii. determine an adaptive distance metric based on the proximity of a new frame to previously selected frames;
• iii. compute a selection score for each frame based on the frame-level uncertainty and the adaptive distance metric;
• iv. select frames for annotation when their selection scores exceed a predefined threshold, ensuring a diverse and informative set of frames is chosen across the temporal domain of the multimedia input;
• c. annotate the selected subset of frames to create a labeled dataset;
• d. use the labeled dataset to train an action detection model to achieve or exceed a predetermined precision benchmark;
• e. update the action detection model iteratively by reapplying the adaptive proximity-aware uncertainty selection model to further refine frame selection based on updated model insights and annotations until the precision benchmark is met or an annotation cost budget is exhausted;
• f. employ an uncertainty-based scoring mechanism that assigns priority to frames with higher uncertainty scores for subsequent annotation;
• g. generate pseudo-labels for non-selected frames utilizing interpolation and a spatio-temporal superpixel method and refine pseudo-labels over time responsive to newly acquired annotations and prior predictions;
• h. assign weights to each frame based on the proximity of their localization to ground-truth annotations, whereby frames closer to the ground-truth data are given higher weight;
• i. adjust the localization loss for each frame based on the assigned weights;
• and
• j. apply the adaptive proximity-aware uncertainty selection model iteratively across the multimedia input.

A method for active sparse labeling of multimedia input to facilitate dense video understanding, implemented by at least one computer processor, the method comprising:

• a. receiving a multimedia input comprising a plurality of frames;
• b. applying an adaptive proximity-aware uncertainty selection model to iteratively select a subset of frames from the multimedia input, the model configured to:
• i. calculate frame-level uncertainty for each frame based on pixel-wise confidence scores of localization;
• ii. determine an adaptive distance metric based on the proximity of a new frame to previously selected frames;
• iii. compute a selection score for each frame based on the frame-level uncertainty and the adaptive distance metric;
• iv. select frames for annotation when their selection scores exceed a predefined threshold, ensuring a diverse and informative set of frames is chosen across the temporal domain of the multimedia input;
• c. annotating the selected subset of frames to create a labeled dataset;
• d. using the labeled dataset to train an action detection model to achieve or exceed a predetermined precision benchmark;
• and
• e. updating the action detection model iteratively by reapplying the adaptive proximity-aware uncertainty selection model to further refine frame selection based on updated model insights and annotations until the precision benchmark is met or an annotation cost budget is exhausted.