# US 2025/0166342 A1 · Monocular 2d semantic keypoint detection and tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166342 A1
- Document ID: US20250166342A1
- Status: Published application
- Kind: A1
- Application number: 19/029,445
- Filed: 2025-01-17
- Published: 2025-05-22
- Earliest claimed: 2021-07-30
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Haofeng CHEN; Arjun BHARGAVA; Rares Andrei AMBRUS; Sudeep PILLAI
- Web page: https://patentgrep.com/patent/US20250166342A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/60: Analysis of geometric attributes
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30236: Traffic on road, railway or crossing
- G06T 2207/30241: Trajectory
- G06T 2207/30248: Vehicle exterior or interior
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for 2D semantic keypoint detection and tracking is described. The method includes learning embedded descriptors of salient object keypoints detected in previous images according to a descriptor embedding space model. The method also includes predicting, using a shared image encoder backbone, salient object keypoints within a current image of a video stream. The method further includes inferring an object represented by the predicted, salient object keypoints within the current image of the video stream. The method also includes tracking the inferred object by matching embedded descriptors of the predicted, salient object keypoints representing the inferred object within the previous images of the video stream based on the descriptor embedding space model.

## Claims

### Claim 1

A method for 2D semantic keypoint detection and tracking, comprising:
generating, by a trained shared descriptor embedding space model, a joint descriptor embedding space, including embedded descriptors of predicted salient object keypoints in a previous frame of a video stream and embedded descriptors of the predicted salient object keypoints within a current frame of the video stream;
matching the embedded descriptors from the joint descriptor embedding space to identify the predicted salient object keypoints within the current frame that correspond with the predicted salient object keypoints within the previous frame;
inferring an object represented by the corresponding predicted, salient object keypoints within the current frame of the video stream; and
tracking the inferred object within the previous frames, the current frame, and subsequent frames of the video stream based on the matching.

### Claim 2 (depends on 1)

The method of claim 1, in which the salient object keypoints comprise salient vehicle keypoints to represent the object as a vehicle.

### Claim 3 (depends on 1)

The method of claim 1, in which learning the embedded descriptors comprises:
learning embedded descriptors of salient vehicle object keypoints detected in previous images to train a shared descriptor embedding space model;
predicting the salient vehicle object keypoints and the descriptors within frames of the video stream;
identifying, using the descriptors, associated keypoints at different frames of the video stream corresponding to the salient vehicle object keypoints based on the trained shared descriptor embedding space model; and
predicting, using a shared image encoder backbone, the salient object keypoints within the current frame of the video stream.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
generating known transformations of an input image;
warping the input image to form a warped image;
extracting keypoints and the descriptors from the input image and the warped image;
computing corresponding keypoints through the known transformations between the input image and the warped image; and
ensuring the descriptors of the corresponding keypoints match the extracted keypoints.

### Claim 5 (depends on 4)

The method of claim 4, in which predicting the salient vehicle object keypoints comprises:
extracting, using the shared image encoder backbone, the salient vehicle object keypoints within the current frame of the video stream based on relevant appearance and geometric features of the current frame; and
generating a keypoint heatmap based on the salient vehicle object keypoints extracted using the shared image encoder backbone.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating, using a descriptor head, the embedded descriptors of the predicted salient object keypoints; and
computing the predicted salient object keypoints in the previous frames of the video stream using the embedded descriptors generated using the descriptor head.

### Claim 7 (depends on 1)

The method of claim 1, in which the object comprises a vehicle represented by salient vehicle object keypoints to approximate geometry/spatial relationships of a rigid-body of the vehicle.

### Claim 8 (depends on 1)

The method of claim 1, further comprising planning a trajectory of an ego vehicle according to the tracking of the inferred object.

### Claim 9

A non-transitory computer-readable medium having program code recorded thereon for 2D semantic keypoint detection and tracking, the program code being executed by a processor and comprising:
program code to generate, by a trained shared descriptor embedding space model, a joint descriptor embedding space, including embedded descriptors of predicted salient object keypoints in a previous frame of a video stream and embedded descriptors of the predicted salient object keypoints within a current frame of the video stream;
program code to match the embedded descriptors from the joint descriptor embedding space to identify the predicted salient object keypoints within the current frame that correspond with the predicted salient object keypoints within the previous frame;
program code to infer an object represented by the corresponding predicted, salient object keypoints within the current frame of the video stream; and
program code to track the inferred object within the previous frames, the current frame, and subsequent frames of the video stream based on the matching.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the salient object keypoints comprise salient vehicle keypoints to represent the object as a vehicle.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to learn the embedded descriptors comprises:
program code to learn embedded descriptors of salient vehicle object keypoints detected in previous images to train a shared descriptor embedding space model;
program code to predict the salient vehicle object keypoints and the descriptors within frames of the video stream;
program code to identify, using the descriptors, associated keypoints at different frames of the video stream corresponding to the salient vehicle object keypoints based on the trained shared descriptor embedding space model; and
program code to predict, using a shared image encoder backbone, the salient object keypoints within the current frame of the video stream.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, further comprising:
program code to generate known transformations of an input image;
program code to warp the input image to form a warped image;
program code to extract keypoints and the descriptors from the input image and the warped image;
program code to compute corresponding keypoints through the known transformations between the input image and the warped image; and
ensuring the descriptors of the corresponding keypoints match the extracted keypoints.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, in which the program code to predict the salient vehicle object keypoints comprises:
program code to extract, using the shared image encoder backbone, the salient vehicle object keypoints within the current frame of the video stream based on relevant appearance and geometric features of the current frame; and
program code to generate a keypoint heatmap based on the salient vehicle object keypoints extracted using the shared image encoder backbone.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising:
program code to generate, using a descriptor head, the embedded descriptors of the predicted salient object keypoints; and
program code to compute the predicted salient object keypoints in the previous frames of the video stream using the embedded descriptors generated using the descriptor head.

### Claim 15 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the object comprises a vehicle represented by salient vehicle object keypoints to approximate geometry/spatial relationships of a rigid-body of the vehicle.

### Claim 16 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising program code to plan a trajectory of an ego vehicle according to the tracking of the inferred object.
