Memory-Guided Video Object Detection

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
• inputting a video into an interleaved object detection model comprising a plurality of feature extractor networks and a shared memory layer, the video comprising a plurality of frames;
• and
• for each of one or more frames:
• selecting one of the plurality of feature extractor networks to analyze the one or more frames;
• analyzing the one or more frames by the selected feature extractor network to determine one or more features of the one or more frames;
• determining an updated set of features based at least in part on the one or more features and one or more previously extracted features extracted from a previous frame stored in the shared memory layer;
• and
• detecting an object in the one or more frames based at least in part on the updated set of features.

A computer-implemented method of training an interleaved object detection model comprising a first feature extractor network, a second feature extractor network, and a shared memory layer, comprising:

• inputting a training video comprising a plurality of image frames into the interleaved object detection model;
• and
• for one or more frames of the video:
• randomly selecting either the first feature extractor network or the second feature extractor network;
• analyzing the one or more image frames of the training video with the selected feature extractor network to determine one or more detected objects present in the plurality of image frames;
• determining a detection loss function based at least in part on a comparison of a ground-truth detection and the one or more detected objects;
• and
• training the interleaved object detection model based at least in part on the detection loss function.