# US 2025/0095167 A1 · Multi-subject multi-camera tracking for high-density environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095167 A1
- Document ID: US20250095167A1
- Status: Published application
- Kind: A1
- Application number: 18/618,974
- Filed: 2024-03-27
- Published: 2025-03-20
- Earliest claimed: 2023-09-19
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Zheng TANG; Sujit BISWAS; Ganapathy Seshadri Cadungude AIYER; Shuo WANG; Akshay AGRAWAL; Sameer Satish PUSEGAONKAR
- Web page: https://patentgrep.com/patent/US20250095167A1

## CPC

- G06T 7/292: Multi-camera tracking
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In various examples, multi-subject multi-camera tracking for high-density environments is provided. In some embodiments, an MTMC tracking system may associate previously initialized behavior states with currently generated behaviors clusters based on selecting between subject trajectory tracking and cluster matching algorithms. The system may receive image data comprising feeds from a plurality of optical images sensors. The system may generate behavior clusters and apply trajectory tracking to determine if the prior behavior states can be propagated based on a continuity of trajectory analysis. If behavior clusters cannot be associated based on trajectory tracking, a matching algorithm may be applied to the set of behavior clusters. For matched clusters, a matched a prior behavior state may be assigned to the cluster and propagated forward. If a cluster does not match with a prior behavior state, a new global ID and behavior state may be initialized based on representations forming the cluster.

## Claims

### Claim 1

One or more processors comprising processing circuitry to:
cluster a plurality of representations corresponding to a behavior of one or more subjects within an environment based on streaming data corresponding to a first time frame to generate one or more clusters based at least on a similarity between representations of the plurality of representations, wherein the streaming data includes behavior data of one or more subjects within the environment generated using a plurality of optical sensors;
determine, based at least on trajectory tracking data for the one or more subjects, whether individual clusters of the one or more clusters are associated with a respective prior behavior state of one or more first behavior states;
assign one or more second behavior states to the one or more clusters based on at least one of the individual clusters of the one or more clusters not having an associated behavior state from the one or more first behavior states; and
update at least one of the one or more first behavior states or the one or more second behavior states based on the plurality of representations to generate updated behavior states.

### Claim 2 (depends on 1)

The one or more processors of claim 1, wherein the one or more second behavior states are assigned to the one or more clusters using a matching algorithm, and wherein the processing circuitry is further to:
initialize a new behavior state for at least one cluster of the one or more clusters based on the at least one cluster not being assigned to at least one of the one or more second behavior states by the matching algorithm.

### Claim 3 (depends on 1)

The one or more processors of claim 1, wherein the behavior data comprises at one or both of appearance data and spatiotemporal data represented by the plurality of representations.

### Claim 4 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to:
generate the plurality of representations based on processing individual data feeds from at least one individual sensor of the plurality of optical sensors.

### Claim 5 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to:
generate the plurality of representations using a machine learning model trained to detect one or more characteristics representing the one or more subjects based on the streaming data.

### Claim 6 (depends on 1)

The one or more processors of claim 1, wherein the streaming data comprises synchronized optical image streaming data that includes individual image feeds from the plurality of optical sensors, wherein at least two optical sensors of the plurality of optical sensors are synchronized to capture the individual image feeds at the same time.

### Claim 7 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to:
map the plurality of representations to a global image coordinate system based at least on camera calibration parameters associated with the plurality of optical sensors.

### Claim 8 (depends on 1)

The one or more processors of claim 1, wherein the plurality of representations individually represent one or both of appearance data and spatiotemporal data for a respective subject of the one or more subjects.

### Claim 9 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to:
associate individual behavior states from at least one of the one or more first behavior states or the one or more second behavior states with a global identifier (ID).

### Claim 10 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to cluster the plurality of representations based on a hierarchical clustering process that performs operations to:
determine a subject prediction number based on a first clustering process that clusters the plurality of representations based at least on a similarity of representations from the plurality of representations; and
apply, to the plurality of representations, a second clustering process that is constrained to cluster the plurality of representations based at least on the subject prediction number.

### Claim 11 (depends on 2)

The one or more processors of claim 2, wherein the matching algorithm comprises at least one of:
an iterative matching combinatorial optimization algorithm;
an algorithm that solves an assignment problem by matching agents to tasks;
a Hungarian matching algorithm;
a Kuhn-Munkres algorithm; or
a Munkres assignment algorithm.

### Claim 12 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to apply a linear programming algorithm to implement overlapping behavior suppression to avoid associating multiple clusters of the one or more clusters with more than one of the one or more subjects.

### Claim 13 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to cause a display of a computer vision-based view of the one or more subjects for at least a portion of the environment based at least on the updated behavior states.

### Claim 14 (depends on 1)

The one or more processors of claim 1, wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for three-dimensional assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 15

A system comprising one or more processors to:
generate one or more clusters using a plurality of representations computed based on optical image streaming data representing one or more tracked subjects within a monitored area;
associate, based at least on trajectory tracking data, individual clusters of the one or more clusters with a respective behavior state of one or more first behavior states computed from the optical image streaming data;
assign, using a matching algorithm, one or more second behavior states to the one or more clusters based on at least one of the individual clusters not having an associated behavior state from the one or more first behavior states; and
update at least one behavior state of the one or more first behavior states or the one or more second behavior states based on behavior data represented by the plurality of representations.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more processors are further to:
generate the plurality of representations using a machine learning model trained to detect one or more characteristics representing the one or more tracked subjects based on the optical image streaming data.

### Claim 17 (depends on 15)

The system of claim 15, wherein the one or more processors are further to cluster the plurality of representations based on a hierarchical clustering process to:
determine a subject prediction number based on a first clustering process that clusters the plurality of representations based at least on a similarity of representations; and
apply, to the plurality of representations, a second clustering process is constrained to cluster the plurality of representations based at least on the subject prediction number.

### Claim 18 (depends on 15)

The system of claim 15, wherein the matching algorithm comprises at least one of:
an iterative matching combinatorial optimization algorithm;
an algorithm that solves an assignment problem by matching agents to tasks;
a Hungarian matching algorithm;
a Kuhn-Munkres algorithm; or
a Munkres assignment algorithm.

### Claim 19 (depends on 15)

The system of claim 15, wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for three-dimensional assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 20

A method comprising:
generating behavior states representing behavior data for one or more tracked subjects within an environment based on clustering a plurality of representations computed from optical image streaming data representing the environment, wherein the behavior states are updated using the plurality of representations by:
associating, based at least on trajectory tracking data, one or more clusters of the plurality of representations with one or more first behavior states computed from the optical image streaming data; and
applying a matching algorithm to assign one or more second behavior states to the one or more clusters based on at least one individual cluster of the plurality of representations not having an associated behavior state from the one or more first behavior states.
