# US 10,121,515 B2 · Method, system and computer program product for interactively identifying same individuals or objects present in video recordings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,121,515 B2
- Document ID: US10121515B2
- Status: Granted
- Kind: B2
- Application number: 15/368,357
- Filed: 2016-12-02
- Published: 2017-12-07
- Granted: 2018-11-06
- Earliest claimed: 2016-06-06
- Assignees: Avigilon Corporation
- Applicants: Avigilon Corporation
- Inventors: Moussa Doumbouya; Mahesh Saptharishi; Eric Sjue; Hannah Valbonesi
- Web page: https://patentgrep.com/patent/US10121515B2

## CPC

- G06Q 10/00: Administration; Management
- G11B 27/22: Means responsive to presence or absence of recorded information signals
- G06T 7/292: Multi-camera tracking
- G06T 7/74: {involving reference images or patches}
- G11B 27/102: {Programmed access in sequence to addressed parts of tracks of operating record carriers (access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816)}
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G11B 27/3081: {used signal is a video-frame or a video-field (P.I.P)}
- G11B 27/309: {Table of contents}
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06T 2207/30241: Trajectory
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method, system and computer program product for interactively identifying same individuals or objects present in video recordings is disclosed. When a thumbnail in a set of thumbnails is selected, new information is obtained. The new information may be that an individual or object is present in the portion of the video recording associated with the thumbnail. A search can be carried out for the individual or object based on the new information. The search generates new match likelihoods for each of displayed thumbnails within a user interface page. The displayed thumbnails are re-ordered based on the new match likelihoods.

## Claims

### Claim 1

A method comprising:
displaying a plurality of sets of thumbnails, each set of thumbnails including one or more thumbnails positioned in a respective row or column which, along with the set of thumbnails, are defined by a unique time interval of a plurality of time intervals, and each thumbnail of each set of thumbnails being visually representative of an associated portion of a video recording taken at the defined time interval for the thumbnail, and the one or more thumbnails in each set of thumbnails being organized in a descending order arrangement starting at one end of the respective row or column with a thumbnail corresponding to a highest likelihood, as compared to any other thumbnails of the set, of appearance of an individual or object having been identified as being of interest, and any remaining thumbnails in the set are displayed, in descending order, extending away from the thumbnail corresponding to the highest likelihood;
allowing selection of a first thumbnail in one of the sets of thumbnails to obtain new information that the individual or object is present in the portion of the video recording associated with the first thumbnail;
carrying out a search for the individual or object based on the new information, wherein the search generates new match likelihoods for each of the displayed thumbnails;
re-ordering the displayed thumbnails based on the new match likelihoods so as to maintain the descending order arrangement in each of the sets of thumbnails.

### Claim 2 (depends on 1)

The method as claimed in claim 1 further comprising displaying new thumbnails interspersed with remaining pre-search thumbnails.

### Claim 3 (depends on 1)

The method as claimed in claim 1 wherein the re-ordering the displayed thumbnails includes re-ordering only a sub-set of pre-search thumbnails with other pre-search thumbnails being no longer displayed due to respectively lowered match likelihoods.

### Claim 4 (depends on 1)

The method as claimed in claim 1 wherein, after the selection of the first thumbnail, the first thumbnail is moved to a user interface space separate from another user interface space within which the plurality of sets of thumbnails are displayed.

### Claim 5 (depends on 1)

The method as claimed in claim 1 wherein the plurality of sets of thumbnails are displayed within a user interface page, and the user interface page includes an interactive tool for changing the plurality of time intervals.

### Claim 6

A tangible, non-transitory, computer-readable storage medium having instructions encoded therein, wherein the instructions, when executed by at least one processor, cause a carrying out of a method comprising:
displaying a plurality of sets of thumbnails, each set of thumbnails including one or more thumbnails positioned in a respective row or column which, along with the set of thumbnails, are defined by a unique time interval of a plurality of time intervals, and each thumbnail of each set of thumbnails being visually representative of an associated portion of a video recording taken at the defined time interval for the thumbnail, and the one or more thumbnails in each set of thumbnails being organized in a descending order arrangement starting at one end of the respective row or column with a thumbnail corresponding to a highest likelihood, as compared to any other thumbnails of the set, of appearance of an individual or object having been identified as being of interest, and any remaining thumbnails in the set are displayed, in descending order, extending away from the thumbnail corresponding to the highest likelihood;
allowing selection of a first thumbnail in one of the sets of thumbnails to obtain new information that the individual or object is present in the portion of the video recording associated with the first thumbnail;
sending a request to a server to search for the individual or object based on the new information, wherein the search generates new match likelihoods for each of the displayed thumbnails;
re-ordering the displayed thumbnails based on the new match likelihoods so as to maintain the descending order arrangement in each of the sets of thumbnails.

### Claim 7 (depends on 6)

The tangible, non-transitory, computer-readable storage medium as claimed in claim 6 wherein the method further comprises displaying new thumbnails interspersed with remaining pre-search thumbnails.

### Claim 8 (depends on 6)

The tangible, non-transitory, computer-readable storage medium as claimed in claim 6 wherein the re-ordering the displayed thumbnails includes re-ordering only a sub-set of pre-search thumbnails with other pre-search thumbnails being no longer displayed due to respectively lowered match likelihoods.

### Claim 9 (depends on 6)

The tangible, non-transitory, computer-readable storage medium as claimed in claim 6 wherein, after the selection of the first thumbnail, the first thumbnail is moved to a user interface space separate from another user interface space within which the plurality of sets of thumbnails are displayed.

### Claim 10 (depends on 6)

The tangible, non-transitory, computer-readable storage medium as claimed in claim 6 wherein the plurality of sets of thumbnails are displayed within a user interface page, and the user interface page includes an interactive tool for changing the plurality of time intervals.