# US 12,183,087 B2 · Semantically aware keypoint matching > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,087 B2 - Document ID: US12183087B2 - Status: Granted - Kind: B2 - Application number: 18/489,687 - Filed: 2023-10-18 - Published: 2024-02-08 - Granted: 2024-12-31 - Earliest claimed: 2020-04-14 - Assignees: Toyota Research Institute, Inc. - Applicants: Toyota Research Institute, Inc. - Inventors: Jiexiong Tang; Rares Andrei Ambrus; Vitor Guizilini; Adrien David Gaidon - Web page: https://patentgrep.com/patent/US12183087B2 ## CPC - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G05D 1/0221: {involving a learning process} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/243: Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G05D 2101/10: using artificial intelligence [AI] techniques - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES ## Abstract A method for keypoint matching performed by a semantically aware keypoint matching model includes generating a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image. The method also includes generating a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semantically segmented image, corresponding to the keypoint. The method further includes controlling an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors. ## Claims ### Claim 1 A method for keypoint matching performed by a semantically aware keypoint matching model, the method comprising: generating a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image; generating a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semanticaly segmented image, corresponding to the keypoint; and controlling an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors. ### Claim 2 (depends on 1) The method of claim 1, further comprising identifying the one or more first augmented keypoint descriptors of the target image matching the one or more second augmented keypoint descriptors of the image based on a nearest neighbor matching function. ### Claim 3 (depends on 2) The method of claim 2, further comprising biasing the matching of the one or more first augmented keypoint descriptors of the target image and the one or more second augmented keypoint descriptors toward two or more descriptors having matching semantic labels. ### Claim 4 (depends on 1) The method of claim 1, further comprising identifying a current location of the agent based on the target image, wherein controlling the action of the agent comprises navigating to a new location from the current location. ### Claim 5 (depends on 1) The method of claim 1, wherein: the target image is identified from a plurality of stored images; and each one of the stored images is associated with one or more augmented keypoint descriptors. ### Claim 6 (depends on 1) The method of claim 1, wherein the agent is an autonomous vehicle or a semi-autonomous vehicle. ### Claim 7 (depends on 1) The method of claim 1, wherein the image is a monocular image. ### Claim 8 An apparatus for keypoint matching performed by a semantically aware keypoint matching model, comprising: one or more processors; and one or more memories coupled with the one or more processors and storing instructions operable, when executed by the one or more processors, to cause the apparatus to: generate a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image; generate a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semanticaly segmented image, corresponding to the keypoint; and control an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein execution of the instructions further cause the apparatus to identify the one or more first augmented keypoint descriptors of the target image matching the one or more second augmented keypoint descriptors of the image based on a nearest neighbor matching function. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein execution of the instructions further cause the apparatus to bias the matching of the one or more first augmented keypoint descriptors of the target image and the one or more second augmented keypoint descriptors toward two or more descriptors having matching semantic labels. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein execution of the instructions further cause the apparatus to identify a current location of the agent based on the target image, wherein controlling the action of the agent comprises navigating to a new location from the current location. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein: the target image is identified from a plurality of stored images; and each one of the stored images is associated with one or more augmented keypoint descriptors. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the agent is an autonomous vehicle or a semi-autonomous vehicle. ### Claim 14 (depends on 8) The apparatus of claim 8, wherein the image is a monocular image. ### Claim 15 A non-transitory computer-readable medium having program code recorded thereon for keypoint matching performed by a semantically aware keypoint matching model, the program code executed by a processor and comprising: program code to generate a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image; program code to generate a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semanticaly segmented image, corresponding to the keypoint; and program code to control an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the program code further comprises program code to identify the one or more first augmented keypoint descriptors of the target image matching the one or more second augmented keypoint descriptors of the image based on a nearest neighbor matching function. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the program code further comprises program code to bias the matching of the one or more first augmented keypoint descriptors of the target image and the one or more second augmented keypoint descriptors toward two or more descriptors having matching semantic labels. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the program code further comprises program code to identify a current location of the agent based on the target image, wherein controlling the action of the agent comprises navigating to a new location from the current location. ### Claim 19 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein: the target image is identified from a plurality of stored images; and each one of the stored images is associated with one or more augmented keypoint descriptors. ### Claim 20 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the agent is an autonomous vehicle or a semi-autonomous vehicle.