# US 11,908,194 B2 · Tracking sparse objects and people in large scale environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,194 B2 - Document ID: US11908194B2 - Status: Granted - Kind: B2 - Application number: 17/203,315 - Filed: 2021-03-16 - Published: 2021-09-16 - Granted: 2024-02-20 - Earliest claimed: 2020-03-16 - Assignees: New York University - Applicants: New York University - Inventors: Yurii S. Piadyk; Carlos Augusto Dietrich; Claudio T Silva - Web page: https://patentgrep.com/patent/US11908194B2 ## CPC - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06F 18/25: Fusion techniques - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 7/292: Multi-camera tracking - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30224: Ball; Puck - G06T 2207/30228: Playing field - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A modular tracking system is described comprising of the network of independent tracking units optionally accompanied by a LIDAR scanner and/or (one or more) elevated cameras. Tracking units are combining panoramic and zoomed cameras to imitate the working principle of the human eye. Markerless computer vision algorithms are executed directly on the units and provide feedback to motorized mirror placed in front of the zoomed camera to keep tracked objects/people in its field of view. Microphones are used to detect and localize sound events. Inference from different sensor is fused in real time to reconstruct high-level events and full skeleton representation for each participant. ## Claims ### Claim 1 A system for tracking sparse objects in an environment, the system comprising: a) a plurality of panoramic cameras, each having a wide field of view; b) a plurality of zoom cameras, each having a narrow field of view which is less than the wide field of view; c) a plurality of adjustment mechanisms for adjusting, for each of the plurality of zoom cameras, the narrow field of view; and d) a controller configured to 1) Receive image information from the plurality of panoramic cameras; 2) Determine a position of at least one of the sparse objects within the environment; and 3) Output a control signal to control the adjustment mechanisms so that at least one of the sparse objects in the environment remains within the narrow field of view of at least one of the plurality of zoom cameras, wherein each of the plurality of adjustment mechanisms is a mirror located within an optical path between at least one of the sparse objects and an imager of a respective one of the zoom cameras, wherein the plurality of panoramic cameras, the plurality of zoom cameras and the plurality of mirrors are distributed in a plurality of units, each of the plurality of units including one of the plurality of panoramic cameras, at least one of the plurality of zoom cameras and at least one of the plurality of mirrors, wherein each of the mirrors is controlled by a motor, and wherein each of the motors controlling the mirrors is driven between exposures of its associated zoom camera. ### Claim 2 (depends on 1) The system of claim 1 wherein the controller is, at least in part, distributed among the plurality of units as a plurality of local computing platforms, and wherein each of the plurality of units further includes at least one of the plurality of local computing platforms. ### Claim 3 (depends on 1) The system of claim 1 wherein each of the motors is a stepper motor and is driven, between exposures of its associated zoom camera, by at least two accurately tinned drive pulses so that the oscillations of the mirror when settling in a final position are reduced and the speed of position adjustment is increased. ### Claim 4 (depends on 1) The system of claim 1 wherein the sparse objects include people and a ball, and wherein the environment is a playing field, rink, or court. ### Claim 5 (depends on 4) The system of claim 4 wherein each of the units is positioned ata height between a height of the playing field, rink, or court, and 3 to 8 feet above the playing field, rink, or court. ### Claim 6 (depends on 2) The system of claim 2, further comprising: e) a sparse object position capturing means for determining the position of at least one of the sparse objects within the playing field or court, wherein each of the units includes a transmitter and receiver, and wherein the controller further includes a centralized portion including 1) A receiver for receiving sparse object positions from the sparse object position capturing means, and image data captured by at least some of the plurality of panoramic cameras and the plurality of zoom cameras, 2) an orchestration module for determining for each of the plurality of units, at least one of the sparse objects to track using (i) the received sparse object positions from the sparse object position capturing means, and (ii) the received image data captured by at least some of the plurality of panoramic cameras and the plurality of zoom cameras, and 3) a transmitter to transmit, to each of the plurality of the units, the determined at least one of the sparse objects to track. ### Claim 7 (depends on 6) The system of claim 6 wherein the transmitter and receiver included in each of the units are wireless, and wherein the transmitter and receiver of the centralized portion of the controller are wireless. ### Claim 8 (depends on 6) The system of claim 6 wherein the sparse object position capturing means is one of (A) a LI DAR unit, and/or (B) an elevated camera, and wherein the sparse object position capturing means covers the entire playing field or court. ### Claim 9 (depends on 1) The system of claim 1 wherein the wide angle field of view is at least 45 degrees, horizontally. ### Claim 10 (depends on 1) The system of claim 1 wherein the narrow view field of view is no more than 45 degrees, horizontally. ### Claim 11 (depends on 1) The system of claim 1 wherein the plurality of panoramic cameras and the plurality of zoom cameras each have an internal clock operating at a first frequency, and wherein the controller is further configured to synchronize the plurality of panoramic cameras and the plurality of zoom cameras by receiving, by a radio module in each of the plurality of panoramic cameras and the plurality of zoom cameras, a common radio synchronization signal; extracting, bya signal extractor in each of the plurality of panoramic cameras and the plurality of zoom cameras, the common radio synchronization signal received; and synchronizing each of the internal clocks using the common radio synchronization signal received and extracted, thereby reducing the drift of the clock. ### Claim 12 (depends on 11) The system of claim 11 wherein, in each of the plurality of panoramic cameras and the plurality of zoom cameras, the radio module and the signal extractor collectively have a deterministic, fixed, time delay which reduces jitter of the synchronized clock. ### Claim 13 (depends on 11) The system of claim 11, wherein the controller is further configured to synchronize the plurality of panoramic cameras and the plurality of zoom cameras by triggering of image capture in each of the plurality of panoramic cameras and the plurality of zoom cameras based on the synchronized internal clocks, wherein a high frame rate is used for zoom cameras and lower frame rate is used for the panoramic cameras. ### Claim 14 (depends on 11) The system of claim 11, further comprising: a microphone in each of the plurality of panoramic cameras and the plurality of zoom cameras, wherein the plurality of panoramic cameras and the plurality of zoom cameras are configured to capture audio, generate streams of audio data, and embed the synchronized internal clocks into the generated streams of audio data to generate modified streams of audio data. ### Claim 15 (depends on 14) The system of claim 14, wherein each of the synchronized internal clocks are embedded into a respective one of the generated streams of audio data either (A) as a completely separate channel of the respective one of the generated streams of audio data, or (B) as a least significant bit in a channel of the respective one of the generated streams of audio data. ### Claim 16 (depends on 15) The system of claim 15, wherein the controller is further configured to synchronize the plurality of panoramic cameras and the plurality of zoom cameras by triangulating a position of an origin of a sound in an environment using the streams of audio data captured by the microphones and the synchronized internal clocks embedded in the modified streams of audio data. ### Claim 17 A computer-implemented method for tracking sparse objects in an environment, the computer-implemented method comprising: a) receiving image information from a plurality of panoramic cameras, each having a wide field of view; b) determining a position of at least one of the sparse objects within the environment; and c) outputting a control signal to control one or more adjustment mechanisms for adjusting, for one or more of a plurality of zoom cameras, each having a narrow field of view which is less than the wide field of view, the narrow field of view so that at least one of the sparse objects in the environment remains within the narrow field of view of one or more of the plurality of zoom cameras, wherein each of the one or more adjustment mechanisms is a mirror located within an optical path between at least one of the objects and an imager of a respective one of the zoom cameras, wherein the plurality of panoramic cameras, the plurality of zoom cameras and the plurality of mirrors are distributed in a plurality of units, each of the plurality of units including one of the plurality of panoramic cameras, at least one of the plurality of zoom cameras and at least one of the plurality of mirrors, wherein each of the mirrors is controlled by a motor, and wherein each of the motors controlling the mirrors is driven between exposures of its associated zoom camera. ### Claim 18 (depends on 1) The system of claim 1 wherein each of the mirrors is controlled by a motor, and wherein each of the motors controlling the mirrors is only driven between exposures of its associated zoom camera whereby oscillations of the mirror and resulting blur during each of the exposures is avoided. ### Claim 19 (depends on 1) The system of claim 1, wherein, for each of the plurality of zoom cameras, a position of the mirror is adjusted so that at least one of the sparse objects is within a field of view of the imager of the zoomed camera.