Semi-supervised 3d object tracking in videos via 2d semantic keypoints

A non-transitory computer-readable medium having program code recorded thereon for 3D object tracking, the program code being executed by a processor and comprising:

• program code to infer first 2D semantic keypoints of a 3D object within a sparsely annotated video stream;
• program code to match the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame;
• program code to warp the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame;
• and
• program code to label a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.

A system for 3D object tracking, the system comprising:

• a 3D object detector module to infer first 2D semantic keypoints of a 3D object within a sparsely annotated video stream;
• keypoint descriptor matching module to match the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame;
• a keypoint flow model to warp the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame;
• and
• a 3D object tracking module to label a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.