Temporal multi-scale clockwork memory networks for object detection in videos

A computer program product for object detection utilizing a clockwork flow guided memory network, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• receiving, by a processor, a plurality of videos, each of the plurality of videos including a plurality of frames;
• generating, by the processor with a feature extraction network, a frame feature map for a current frame of the plurality of frames;
• aggregating, by the processor, a memory feature map determined from the frame feature map and previous memory feature maps from previous frames on a plurality of time axes by warping the previous memory feature maps, with the plurality of time axes including a first time axis at a first frame increment and a second time axis at a second frame increment with the second frame increment greater than the first frame increment, the warping being updated at each of the plurality of frames and being configured to perform a single-warp computation per-frame;
• predicting, by the processor with a task network, an object from the memory feature map;
• and
• controlling an operation of a processor-based machine to react in accordance with the object.

An object detector that detects objects in video, the object detector comprising:

• a processing system including a processor device and memory coupled to the processor device, the processing system having program code configured to:
• receive a plurality of videos, each of the plurality of videos including a plurality of frames;
• generate, with a feature extraction network, a frame feature map for a current frame of the plurality of frames;
• aggregate a memory feature map determined from the frame feature map and previous memory feature maps from previous frames on a plurality of time axes by warping the previous memory feature maps, with the plurality of time axes including a first time axis at a first frame increment and a second time axis at a second frame increment with the second frame increment greater than the first frame increment, the warping being updated at each of the plurality of frames and being configured to perform a single-warp computation per-frame;
• predict, with a task network, an object from the memory feature map;
• and
• control an operation of a processor-based machine to react in accordance with the object.