# US 11,907,293 B2 · Reasoning from surveillance video via computer vision-based multi-object tracking and spatiotemporal proximity graphs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,293 B2 - Document ID: US11907293B2 - Status: Granted - Kind: B2 - Application number: 17/120,349 - Filed: 2020-12-14 - Published: 2022-06-16 - Granted: 2024-02-20 - Earliest claimed: 2020-12-14 - Assignees: CACI, Inc.—Federal - Applicants: CACI, Inc.—Federal - Inventors: Zachary Jorgensen; Tyler Staudinger; Charles Viss - Web page: https://patentgrep.com/patent/US11907293B2 ## CPC - G06F 16/787: using geographical or spatial information, e.g. location - G06F 16/75: Clustering; Classification - G06F 16/784: {the detected or recognised objects being people} - G06F 16/786: {using motion, e.g. object motion or camera motion} - G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)} - G06F 40/56: Natural language generation - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06V 20/47: {Detecting features for summarising video content} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06N 3/045: Combinations of networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems, and apparatuses, among other things, may detect and store activity in videos based on a spatiotemporal graph representation. Spatiotemporal proximity graphs may be built based on one or more received tracks and may include one or more nodes and each node may include one or more attributes associated with a corresponding entity. One or more spatiotemporal relationships may be identified between the entities based on each spatiotemporal proximity graph one or more activities of the entities may be identified based on the spatiotemporal relationships. ## Claims ### Claim 1 A method comprising: receiving a scene including a plurality of tracks, each track of the plurality of tracks corresponding to an entity and each track of the plurality of tracks being associated with a bounding box; determining, based on a distance between a plurality of the bounding boxes, a defined spatial proximity between the plurality of the bounding boxes; building a spatiotemporal proximity graph based on the plurality of tracks; and identifying, based on the spatiotemporal proximity graph and the spatial proximity, a spatiotemporal relationship between a plurality of the entities. ### Claim 2 (depends on 1) The method of claim 1, further comprising identifying a node for each entity, wherein each node includes an attribute indicating a time interval in which the node was present in the scene and the spatiotemporal proximity graph includes the identified nodes. ### Claim 3 (depends on 2) The method of claim 2, wherein each node includes a second attribute indicating a track identifier associated with the corresponding track. ### Claim 4 (depends on 1) The method of claim 1, further comprising identifying one or more proximity edges, wherein each proximity edge includes an attribute indicating a span of frames in which two entities of the plurality of entities are within the defined spatiotemporal proximity. ### Claim 5 (depends on 1) The method of claim 1, wherein the defined spatiotemporal proximity is further based on a threshold distance between the bounding boxes for the tracks associated with the plurality of the entities. ### Claim 6 (depends on 1) The method of claim 1, further comprising identifying one or more activities of the plurality of entities based on the identified spatiotemporal relationship. ### Claim 7 (depends on 6) The method of claim 6, further comprising identifying one or more complex activities based on the identified one or more activities. ### Claim 8 (depends on 6) The method of claim 6, further comprising building an activity graph based on the identified one or more activities of the plurality of entities. ### Claim 9 A computer program product comprising a non-transitory computer-readable storage medium; and instructions stored on the non-transitory computer-readable storage medium that, when executed by a processor, causes the processor to: receive overhead video footage; receive a scene associated with the overhead video footage including a plurality of tracks, each track of the plurality of tracks corresponding to an entity and each track of the plurality of tracks being associated with a bounding box; build a spatiotemporal proximity graph based on the plurality of tracks; identify, based on the spatiotemporal proximity graph and a distance between a plurality of the entities, a spatiotemporal relationship between the plurality of the entities; and identify one or more portions of the overhead video footage that correspond to the spatiotemporal proximity graph. ### Claim 10 (depends on 9) The non-transitory computer program product of claim 9, wherein the instructions further cause the processer to identify a node for each entity, wherein each node includes an attribute indicating a time interval in which the node was present in the scene and the spatiotemporal proximity graph includes the identified nodes. ### Claim 11 (depends on 9) The non-transitory computer program product of claim 9, wherein the overhead video footage is received from an aerial vehicle. ### Claim 12 (depends on 9) The non-transitory computer program product of claim 9, wherein building the spatiotemporal proximity graph is based on a flight path of an aerial vehicle.