Incremental map building using learnable features and descriptors

An apparatus for localization performed at an agent, comprising:

• a processor;
• a memory coupled with the processor;
• and
• instructions stored in the memory and operable, when executed by the processor, to cause the apparatus to:
• capture, via one or more sensors integrated with the vehicle, a query image of a current environment of the vehicle, the query image being a two-dimensional (2D) image;
• identify a target image, from a plurality of images associated with a three-dimensional (3D) map of the current environment, comprising a first set of keypoints that match a second set of keypoints of the query image, each image of the plurality of images associated with a respective set of keypoints that are labelled via a keypoint model trained for 2D-to-3D keypoint matching;
• determine a current location within the 3D map based on identifying the target image;
• and
• autonomously or semi-autonomously navigate through the current environment based on determining the current location.

A non-transitory computer-readable medium having program code recorded thereon for localization performed at an agent, the program code executed by a processor and comprising:

• program code to capture, via one or more sensors integrated with the vehicle, a query image of a current environment of the vehicle, the query image being a two-dimensional (2D) image;
• program code to identify a target image, from a plurality of images associated with a three-dimensional (3D) map of the current environment, comprising a first set of keypoints that match a second set of keypoints of the query image, each image of the plurality of images associated with a respective set of keypoints that are labelled via a keypoint model trained for 2D-to-3D keypoint matching;
• program code to determine a current location within the 3D map based on identifying the target image;
• and
• program code to autonomously or semi-autonomously navigate through the current environment based on determining the current location.