Semantically aware keypoint matching

An apparatus for keypoint matching performed by a semantically aware keypoint matching model, comprising:

• one or more processors;
• and
• one or more memories coupled with the one or more processors and storing instructions operable, when executed by the one or more processors, to cause the apparatus to:
• generate a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image;
• generate a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semanticaly segmented image, corresponding to the keypoint;
• and
• control an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors.

A non-transitory computer-readable medium having program code recorded thereon for keypoint matching performed by a semantically aware keypoint matching model, the program code executed by a processor and comprising:

• program code to generate a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image;
• program code to generate a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semanticaly segmented image, corresponding to the keypoint;
• and
• program code to control an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors.