Video representation of first-person videos for activity recognition without labels

A computer-implemented method for activity recognition, the method comprising:

• receiving, by a processor, a plurality of videos, the plurality of videos including labeled videos and unlabeled videos;
• extracting, by the processor with a feature extraction convolutional neural network (CNN), frame features for frames from each of the plurality of videos;
• estimating, by the processor with a feature aggregation system, a vector representation for one of the plurality of videos responsive to the frame features;
• classifying, by the processor, an activity from the vector representation;
• and
• controlling an operation of a processor-based machine to change a state of the processor-based machine, responsive to the activity;
• said method further including a domain adversarial training system wherein the domain adversarial training system employs a video-wise discriminator to learn differences between the labeled videos and the unlabeled videos at the video level after the estimating and to backpropagate the differences into the feature extraction CNN and the feature aggregation system.

A computer program product for activity recognition, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• receiving, by a processor, a plurality of videos, the plurality of videos including labeled videos and unlabeled videos;
• extracting, by the processor with a feature extraction convolutional neural network (CNN), frame features for frames from each of the plurality of videos;
• estimating, by the processor with a feature aggregation system, a vector representation for one of the plurality of videos responsive to the frame features;
• classifying, by the processor, an activity from the vector representation;
• and
• controlling an operation of a processor-based machine to change a state of the processor-based machine, responsive to the activity;
• wherein the feature aggregation system determines an average of the frame features, determines a weighted average of the frame features responsive to a weighting factor, and employs both a short-term memory neural network and a gated recurrent unit network.