Techniques for robust real-time multiple-object tracking with detection propagation and per-class optimization

A method implemented in a data processing system for tracking multiple objects, the method comprising:

• obtaining a frame of video content at an object detection pipeline, the video content comprising a plurality of frames;
• analyzing the frame of video content using an object detection model to detect a plurality of objects in the frame of video content, the object detection model associating each object of the plurality of objects with a confidence score;
• determining whether to perform object detection on the frame or detection propagation based on whether performing the object detection on the frame would cause a frame rate of the object detection pipeline to fall below a threshold;
• responsive to determining that the frame rate would fall below the threshold if the object detection pipeline were to perform object detection on the frame:
• performing detection propagation to extrapolate object tracks for a plurality of objects from previously determined object tracks;
• and
• outputting the object tracks for the plurality of objects for the frame;
• and
• responsive to determining that the frame rate would not fall below the threshold,
• performing the object detection on the frame by:
• performing a primary matching operation on high confidence detection objects to determine first object tracks of the high confidence detection objects by associating the high confidence detection objects with an object track of a plurality of object tracks, the high confidence detection objects being objects from the plurality of objects associated with a confidence score that satisfies a confidence threshold, the first object tracks tracking the high confidence detection objects across the plurality of frames;
• performing a secondary matching operation on low confidence detection objects to determine second object tracks of the low confidence detection objects by associating the low confidence detection objects with an object track of the plurality of object tracks, the low confidence detection objects being objects from the plurality of objects associated with a confidence score that does not satisfy the confidence threshold, the second object tracks tracking the low confidence detection objects across the plurality of frames;
• and
• outputting, from the object detection pipeline, the first object tracks and the second object tracks.

A data processing system comprising:

• a processor;
• and
• a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
• obtaining a frame of video content at an object detection pipeline, the video content comprising a plurality of frames;
• analyzing the frame of video content using an object detection model to detect a plurality of objects in the frame of video content, the object detection model associating each object of the plurality of objects with a confidence score;
• determining whether performing object detection on the frame would cause a frame rate of the object detection pipeline to fall below a threshold;
• responsive to determining that performing the object detection would not cause the frame rate to fall below the threshold, performing object detection and tracking comprising:
• performing a primary matching operation on high confidence detection objects to determine first object tracks of the high confidence detection objects by associating the high confidence detection objects with an object track of a plurality of object tracks, the high confidence detection objects being objects from the plurality of objects associated with a confidence score that satisfies a confidence threshold, the first object tracks tracking the high confidence detection objects across the plurality of frames, and
• performing a secondary matching operation on low confidence detection objects to determine second object tracks of the low confidence detection objects by associating the low confidence detection objects with an object track of the plurality of object tracks, the low confidence detection objects being objects from the plurality of objects associated with a confidence score that does not satisfy the confidence threshold, the second object tracks tracking the low confidence detection objects across the plurality of frames;
• responsive to determining that performing the object detection would cause the frame rate to fall below the threshold, performing detection propagation to extrapolate object tracks for the plurality of objects from previously determined object tracks to extend the first object tracks and the second object tracks;
• and
• outputting, from the object detection pipeline, the first object tracks and the second object tracks.