# US 2025/0166210 A1 · Techniques for single-object tracking in the real world with hierarchical appearance modeling and verification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166210 A1
- Document ID: US20250166210A1
- Status: Published application
- Kind: A1
- Application number: 18/515,487
- Filed: 2023-11-21
- Published: 2025-05-22
- Earliest claimed: 2023-11-21
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Jay PATRAVALI; Rishi MADHOK; Nikolaos KARIANAKIS; Peng LEI
- Web page: https://patentgrep.com/patent/US20250166210A1

## CPC

- G06T 7/248: {involving reference images or patches}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 7/11: Region-based segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A data processing system implements obtaining video content including a target object to be tracked and an object template providing a representation of the target object; analyzing the frames of the video content and the object templates using a single object tracking (SOT) pipeline configured to identify a position of the target object in the frames of the video content, the SOT pipeline outputting a first tracking results; analyzing the frames of the video content using a multiple object tracking (MOT) pipeline that tracks positions of multiple objects in the frames of the video content, the multiple objects including the tracked object and one or more distractor objects, the MOT pipeline outputting second tracking results; comparing the first and second tracking results to determine whether the results are consistent; and tracking the target object using the first tracking results responsive to results being consistent.

## Claims

### Claim 1

A data processing system comprising:
a processor; and
a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising:
obtaining video content that includes a target object to be tracked across frames of the video content and an object template providing a representation of the target object;
analyzing the frames of the video content and the object templates using a single object tracking (SOT) pipeline that analyzes the frames of the video content and the object template with a first machine learning model trained to identify a position of the target object in the frames of the video content, the SOT pipeline outputting a first tracking results;
analyzing the frames of the video content using a multiple object tracking (MOT) pipeline that analyzes the frames of the video content using a second machine learning model trained to track positions of multiple objects in the frames of the video content, the multiple objects including the tracked object and one or more distractor objects, the MOT pipeline outputting second tracking results;
comparing the first tracking results and the second tracking results to determine whether the first tracking results are consistent with the second tracking results; and
tracking the target object using the first tracking results responsive to the first tracking results being consistent with the second tracking results.

### Claim 2 (depends on 1)

The data processing system of claim 1, wherein the first machine learning model is a discriminative classifier model.

### Claim 3 (depends on 1)

The data processing system of claim 1, wherein the second machine learning model is implementing using a DeepSORT architecture.

### Claim 4 (depends on 1)

The data processing system of claim 1, wherein the first tracking results include a first bounding box associated with the target object, wherein the second tracking results include a second bounding box associated with the target object, and wherein comparing the first tracking results and the second tracking results to determine whether the first tracking results are consistent with the second tracking results further comprises:
comparing the first bounding box with the second bounding box to determine an overlap between the first bounding box and the second bounding box; and
determining that the first tracking results are consistent with the second tracking results responsive to the overlap between the first bounding box and the second bounding box satisfying a similarity threshold.

### Claim 5 (depends on 1)

The data processing system of claim 1, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
obtaining a user annotation on a from the frame of the video content identifying the target object; and
extracting the object template from the frame of the video content.

### Claim 6 (depends on 1)

The data processing system of claim 1, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
tracking the target object using a linear motion model responsive to the first tracking results not being consistent with the second tracking results.

### Claim 7 (depends on 6)

The data processing system of claim 6, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
performing a reidentification operation responsive to the first tracking results not being consistent with the second tracking results.

### Claim 8 (depends on 7)

The data processing system of claim 7, wherein performing the reidentification operation responsive to the first tracking results not being consistent with the second tracking results further comprises:
comparing features of the tracked object from the first tracking results with previously captured features of the tracked object stored in a hierarchical feature gallery to determine a similarity score representing a similarity between the tracked object from the first tracking results and previously captured features of the tracked object; and
determining a status of the tracked object based on the similarity score.

### Claim 9 (depends on 8)

The data processing system of claim 8, wherein the hierarchical feature gallery includes a short-term appearance feature bank that stores images of the tracked object from a predetermined number of most recently processed frames from the frames of the video content, and a long-term appearance feature bank that includes all features of the tracked object for all frames of the video content processed while tracking the tracked object.

### Claim 10 (depends on 8)

The data processing system of claim 8, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
analyzing a next frame of the frames of the video content to track the tracked object responsive to the status indicating that the tracked object is missing or occluded.

### Claim 11 (depends on 8)

The data processing system of claim 8, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
tracking the target object using the first tracking results responsive to the status indicating that the first tracking results are correct.

### Claim 12 (depends on 8)

The data processing system of claim 8, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
analyzing a next frame of the frames of the video content to track the tracked object responsive to the status indicating that the tracked object is lost.

### Claim 13

A method implemented in a data processing system for tracking objects in video content, the method comprising:
obtaining video content that includes a target object to be tracked across frames of the video content and an object template providing a representation of the target object;
analyzing the frames of the video content and the object templates using a single object tracking (SOT) pipeline that analyzes the frames of the video content and the object template with a first machine learning model trained to identify a position of the target object in the frames of the video content, the SOT pipeline outputting a first tracking results;
analyzing the frames of the video content using a multiple object tracking (MOT) pipeline that analyzes the frames of the video content using a second machine learning model trained to track positions of multiple objects in the frames of the video content, the multiple objects including the tracked object and one or more distractor objects, the MOT pipeline outputting second tracking results;
comparing the first tracking results and the second tracking results to determine whether the first tracking results are consistent with the second tracking results; and
tracking the target object using the first tracking results responsive to the first tracking results being consistent with the second tracking results.

### Claim 14 (depends on 13)

The method of claim 13, wherein the first machine learning model is a discriminative classifier model.

### Claim 15 (depends on 13)

The method of claim 13, wherein the second machine learning model is implementing using a DeepSORT architecture.

### Claim 16 (depends on 1)

The data processing system of claim 1, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
obtaining a user annotation on a from the frame of the video content identifying the target object; and
extracting the object template from the frame of the video content.

### Claim 17 (depends on 13)

The method of claim 13, wherein the first tracking results include a first bounding box associated with the target object, wherein the second tracking results include a second bounding box associated with the target object, and wherein comparing the first tracking results and the second tracking results to determine whether the first tracking results are consistent with the second tracking results further comprises:
comparing the first bounding box with the second bounding box to determine an overlap between the first bounding box and the second bounding box; and
determining that the first tracking results are consistent with the second tracking results responsive to the overlap between the first bounding box and the second bounding box satisfying a similarity threshold.

### Claim 18

A data processing system comprising:
a processor; and
a machine-readable medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising:
obtaining video content that includes a target object to be tracked across frames of the video content and an object template providing a representation of the target object;
analyzing the frames of the video content and the object templates using a single object tracking (SOT) pipeline that analyzes the frames of the video content and the object template with a first machine learning model trained to identify a position of the target object in the frames of the video content, the SOT pipeline outputting a first tracking results;
analyzing the frames of the video content using a multiple object tracking (MOT) pipeline that analyzes the frames of the video content using a second machine learning model trained to track positions of multiple objects in the frames of the video content, the multiple objects including the tracked object and one or more distractor objects, the MOT pipeline outputting second tracking results;
comparing the first tracking results and the second tracking results to determine whether the first tracking results are consistent with the second tracking results; and
reidentifying the tracked object using a hierarchical feature gallery responsive to the first tracking results not being consistent with the second tracking results, wherein the hierarchical feature gallery includes short-term appearance feature bank that stores images of the tracked object from a predetermined number of most recently processed frames from the frames of the video content, and a long-term appearance feature bank that includes all features of the tracked object for all frames of the video content processed while tracking the tracked object.

### Claim 19 (depends on 18)

The data processing system of claim 18, wherein the first tracking results include a first bounding box associated with the target object, wherein the second tracking results include a second bounding box associated with the target object, and wherein comparing the first tracking results and the second tracking results to determine whether the first tracking results are consistent with the second tracking results further comprises:
comparing the first bounding box with the second bounding box to determine an overlap between the first bounding box and the second bounding box; and
determining that the first tracking results are consistent with the second tracking results responsive to the overlap between the first bounding box and the second bounding box satisfying a similarity threshold.

### Claim 20 (depends on 18)

The data processing system of claim 18, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
obtaining a user annotation on a frame of the frames of the video content identifying the target object; and
extracting the object template from the frame of the video content.
