Self-supervised 3D keypoint learning for ego-motion estimation

A non-transitory computer-readable medium having program code recorded thereon for learning depth-aware keypoints and associated descriptors from monocular video, the program code being executed by a processor and comprising:

• program code to train a keypoint network and a depth network to learn the depth-aware keypoints and the associated descriptors based on a target image and a context image from successive images of the monocular video;
• program code to lift 2D keypoints from the target image to learn first 3D keypoints of the target image based on a learned depth map of the target image from the depth network;
• program code to estimate a pose transformation from the target image to the context image based on the first 3D keypoints of the target image to geometrically match the first 3D keypoints in the target image to first 3D keypoints in the context image;
• program code to compute second 3D keypoints of the target image based on an associated ego-motion computed between the target image and the context image according to the first 3D keypoints of the target image and the first 3D keypoints of the context image;
• program code to estimate ego-motion of an ego vehicle from the target image to the context image captured by the ego vehicle based on the second 3D keypoints in the target image and the first 3D keypoints in the context image;
• and
• program code to plan a trajectory of the ego vehicle according to the estimated ego-motion of the ego vehicle.

A system for learning depth-aware keypoints and associated descriptors from monocular video, the system comprising:

• a depth-aware keypoint model trained to learn a keypoint network and a depth network, and trained to learn the depth-aware keypoints and the associated descriptors based on a target image and a context image from successive images of the monocular video;
• a keypoint lifting module to lift 2D keypoints from the target image to learn first 3D keypoints of the target image based on a learned depth map of the target image from the depth network;
• a pose estimation module to estimate a pose transformation from the target image to the context image based on the first 3D keypoints of the target image to geometrically match the first 3D keypoints in the target image to first 3D keypoints in the context image and to compute second 3D keypoints of the target image based on an associated ego-motion computed between the target image and the context image according to the first 3D keypoints of the target image and the first 3D keypoints of the context image;
• an ego-motion estimation module to estimate ego-motion of an ego vehicle from a target image to a context image captured by the ego vehicle based on the second 3D keypoints in the target image and the first 3D keypoints in the context image;
• and
• a planner module to plan a trajectory of the ego vehicle according to the estimated ego-motion of the ego vehicle.