# US 2022/0284598 A1 · Semi-supervised 3d object tracking in videos via 2d semantic keypoints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284598 A1
- Document ID: US20220284598A1
- Status: Published application
- Kind: A1
- Application number: 17/195,436
- Filed: 2021-03-08
- Published: 2022-09-08
- Earliest claimed: 2021-03-08
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Arjun BHARGAVA; Sudeep PILLAI; Kuan-Hui LEE
- Web page: https://patentgrep.com/patent/US20220284598A1

## CPC

- G06T 7/248: {involving reference images or patches}
- G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces
- G01S 13/726: {Multiple target tracking}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/931: of land vehicles
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/931: of land vehicles
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- G05D 1/0253: {extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70)}
- G05D 1/228: Command input arrangements located on-board unmanned vehicles
- G05D 1/24: Arrangements for determining position or orientation
- G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80)
- G06N 3/045: Combinations of networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/48: {Matching video sequences}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G01S 15/931: of land vehicles
- G01S 17/66: Tracking systems using electromagnetic waves other than radio waves
- G01S 2013/9316: {combined with communication equipment with other vehicles or with base stations}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30261: Obstacle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for 3D object tracking is described. The method includes inferring first 2D semantic keypoints of a 3D object within a sparsely annotated video stream. The method also includes matching the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame. The method further includes warping the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame. The method also includes labeling a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.

## Claims

### Claim 1

A method for 3D object tracking, comprising:
inferring first 2D semantic keypoints of a 3D object within a sparsely annotated video stream;
matching the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame;
warping the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame; and
labeling a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.

### Claim 2 (depends on 1)

The method of claim 1, in which labeling a 3D bounding box further comprises annotating the 3D bounding box in the next frame according to a label of a ground truth 3D bounding box in the current frame of the sparsely annotated video stream.

### Claim 3 (depends on 1)

The method of claim 1, in which inferring the first 2D semantic keypoints comprises:
linking the 2D semantic keypoints of the 3D object within the current frame of the sparsely annotated video stream into a 2D structured object geometry corresponding to a category of the 3D object;
embedding descriptors within the 2D semantic keypoints of the 2D structured object geometry; and
lifting the linked, 2D semantic keypoints of the 2D structured object geometry to a 3D structured object geometry.

### Claim 4 (depends on 3)

The method of claim 3, in which linking comprises:
extracting, using a shared image backbone, interest keypoints within a first image of the sparsely annotated video stream based on relevant appearance and geometric features of the first image; and
generating a keypoint heatmap based on the extracted interest keypoints.

### Claim 5 (depends on 3)

The method of claim 3, in which embedding comprises:
generating descriptors of the first 2D semantic keypoints; and
embedding, using a descriptor head, the generated descriptors within the linked, 2D semantic keypoints of the 2D structured object geometry.

### Claim 6 (depends on 1)

The method of claim 1, further comprising tracking the 3D object within the next frame of the sparsely annotated video stream using the embedded descriptors within the 2D semantic keypoints of the 3D structured object geometry.

### Claim 7 (depends on 6)

The method of claim 6, in which tracking comprises:
generating an estimated 2D structured object geometry corresponding to a category of the object in the next frame of the video stream according to the embedded descriptors within the linked 2D semantic keypoints of the 2D structured object geometry corresponding to the 3D object in a first image and a next 2D structured object geometry corresponding to the category of the object in the next frame of the video stream; and
performing tracking loss of the estimated 2D graph structure by comparing the estimated 2D graph structure of the object in the next frame with the generated 2D graph structure of the object in the current frame.

### Claim 8 (depends on 6)

The method of claim 6, further comprising planning a trajectory of an ego vehicle according to the tracking of the object.

### Claim 9

A non-transitory computer-readable medium having program code recorded thereon for 3D object tracking, the program code being executed by a processor and comprising:
program code to infer first 2D semantic keypoints of a 3D object within a sparsely annotated video stream;
program code to match the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame;
program code to warp the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame; and
program code to label a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to label a 3D bounding box further comprises program code to annotate the 3D bounding box in the next frame according to a label of a ground truth 3D bounding box in the current frame of the sparsely annotated video stream.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to infer the first 2D semantic keypoints comprises:
program code to link the 2D semantic keypoints of the 3D object within the current frame of the sparsely annotated video stream into a 2D structured object geometry corresponding to a category of the 3D object;
program code to embed descriptors within the 2D semantic keypoints of the 2D structured object geometry; and
program code to lift the linked, 2D semantic keypoints of the 2D structured object geometry to a 3D structured object geometry.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, in which the program code to link comprises:
program code to extract, using a shared image backbone, interest keypoints within a first image of the sparsely annotated video stream based on relevant appearance and geometric features of the first image; and
program code to generate a keypoint heatmap based on the extracted interest keypoints.

### Claim 13 (depends on 11)

The non-transitory computer-readable medium of claim 11, in which the program code to embed comprises:
program code to generate the descriptors of the first 2D semantic keypoints; and
program code to embed, using a descriptor head, the generated descriptors within the linked, 2D semantic keypoints of the 2D structured object geometry.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising program code to track the 3D object within the next frame of the sparsely annotated video stream using the embedded descriptors within the 2D semantic keypoints of the 3D structured object geometry.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, in which the program code to track comprises:
program code to generate an estimated 2D structured object geometry corresponding to a category of the object in the next frame of the video stream according to the embedded descriptors within the linked 2D semantic keypoints of the 2D structured object geometry corresponding to the 3D object in a first image and a next 2D structured object geometry corresponding to the category of the object in the next frame of the video stream; and
program code to perform tracking loss of the estimated 2D graph structure by comparing the estimated 2D graph structure of the object in the next frame with the generated 2D graph structure of the object in the current frame.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, further comprising program code to plan a trajectory of an ego vehicle according to the tracking of the object.

### Claim 17

A system for 3D object tracking, the system comprising:
a 3D object detector module to infer first 2D semantic keypoints of a 3D object within a sparsely annotated video stream;
keypoint descriptor matching module to match the first 2D semantic keypoints of a current frame with second 2D semantic keypoints in a next frame of the sparsely annotated video stream using embedded descriptors within the current frame and the next frame;
a keypoint flow model to warp the first 2D semantic keypoints to the second 2D semantic keypoints to form warped 2D semantic keypoints in the next frame; and
a 3D object tracking module to label a 3D bounding box in the next frame according to the warped 2D semantic keypoints in the next frame.

### Claim 18 (depends on 17)

The system of claim 17, in which the 3D object tracking module is configured to label the 3D bounding box by annotating the 3D bounding box in the next frame according to a label of a ground truth 3D bounding box in the current frame of the sparsely annotated video stream.

### Claim 19 (depends on 17)

The system of claim 17, in which the 3D object tracking module is configured to track the 3D object within the next frame of the sparsely annotated video stream using the embedded descriptors within the 2D semantic keypoints of the 3D structured object geometry.

### Claim 20 (depends on 17)

The system of claim 17, further comprising a planner module to plan a trajectory of an ego vehicle according to the tracking of the 3D object.
