# US 2009/0154806 A1 · Temporal segment based extraction and robust matching of video fingerprints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2009/0154806 A1
- Document ID: US20090154806A1
- Status: Published application
- Kind: A1
- Application number: 11/958,298
- Filed: 2007-12-17
- Published: 2009-06-18
- Earliest claimed: 2007-12-17
- Assignees: Sinoeast Concept Limited
- Inventors: Jane Wen Chang; Apostol Ivanov Natsev; John R. Smith
- Web page: https://patentgrep.com/patent/US20090154806A1

## CPC

- G06V 20/48: {Matching video sequences}
- G06F 16/70: of video data
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H04H 60/37: for identifying segments of broadcast information, e.g. scenes or extracting programme ID
- H04H 60/59: of video {(recognising characters or patterns in general G06F18/00, G06V20/00)}
- H04H 2201/90: characterised by the use of signatures
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)

## Abstract

A computer implemented method, apparatus, and computer program product code for temporal, event-based video fingerprinting. In one embodiment, events in video content are detected. The video content comprises a plurality of video frames. An event represents discrete points of interest in the video content. A set of temporal, event-based segments are generated using the events. Each temporal, event-based segment is a segment of the video content covering a set of events. A time series signal is derived from each temporal, event-based segment using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment. A temporal segment based fingerprint is extracted based on the time series signal for the each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content.

## Claims

### Claim 1

A computer implemented method for temporal, event-based video fingerprinting, the computer implemented method comprising:
detecting events in video content, wherein the video content comprises a plurality of video frames, and wherein an event represents discrete points of interest in the video content;
generating a set of temporal, event-based segments using the events, wherein each temporal, event-based segment is a segment of the video content covering a set of events;
deriving a time series signal from each temporal, event-based segment in the set of temporal, event-based segments using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment; and
extracting a segment fingerprint based on the time series signal for the each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content.

### Claim 2 (depends on 1)

The computer implemented method of claim 1 wherein the set of temporal segment based fingerprints are used to determine if a portion of a first video clip is derived from a same original content as a second video clip, the computer implemented method further comprising:
comparing temporal segment based fingerprints for the first video clip with temporal segment based fingerprints generated for the second video clip;
identifying matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;
collecting all matching segments between the first video clip and the second video clip to form a set of matching segments;
selecting a subset of matching segments, wherein the subset of matching segments comprises matching segments associated with the first video clip that produces a good linear fit to matching segments associated with the second video clip;
identifying an overall video match score for the first video clip and the second video clip based on the selected matching segments; and
determining whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

### Claim 3 (depends on 2)

The computer implemented method of claim 2 further comprising:
comparing the overall video match score to a threshold score; and
responsive to the overall video match score exceeding the threshold score, identifying the first video clip as a near-duplicate of the second video clip.

### Claim 4 (depends on 1)

The computer implemented method of claim 1 wherein the temporal segment based fingerprints are used for at least one of content-based search, concept detection, content categorization, summarization, filtering, routing, or targeted advertising.

### Claim 5 (depends on 1)

The computer implemented method of claim 1 wherein the content-based features comprises at least one of audio features and visual features, and wherein each event is detected based on at least one of video shot detection, scene changes, speaker changes, audio changes, frame intensity changes, or changes based on low-level content-based descriptors of color, texture, shape, edges, or motion.

### Claim 6 (depends on 1)

The computer implemented method of claim 1 wherein the temporal, event-based segments span a set of frames in the plurality of frames, and wherein the set of frames covers only a subset of the video content.

### Claim 7 (depends on 1)

The computer implemented method of claim 1 wherein a first temporal, event-based segment associated with the video content overlaps with a second temporal, event-based segment associated with the video content.

### Claim 8 (depends on 1)

The computer implemented method of claim 1 wherein the time series signal is based on at least one of temporal tracking of overall frame intensity, tracking of frame region-based intensity sequences, tracking of grid-based intensity sequences, and tracking of adjacent frame differences.

### Claim 9 (depends on 1)

The computer implemented method of claim 1 wherein the temporal segment based fingerprints are extracted from the segment time series based on at least one of uniform sampling, piece-wise linear approximation, Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT), and Discrete Fourier Transform (DFT).

### Claim 10 (depends on 1)

The computer implemented method of claim 1 wherein a temporal segment based fingerprint comprises a fixed-dimensionality feature vector.

### Claim 11 (depends on 1)

The computer implemented method of claim 1 wherein an event describes a significant change of state in at least one of audio content of the video, visual content of the video, and semantic content of the video.

### Claim 12

A computer program product for temporal, event-based video fingerprinting, the computer program product comprising:
a computer readable medium;
program code stored on the computer readable medium for detecting events in video content, wherein the video content comprises a plurality of video frames, and wherein an event represents discrete points of interest in the video content;
program code stored on the computer readable medium for generating a set of temporal, event-based segments using the events, wherein each temporal, event-based segment is a segment of the video content covering a set of events;
program code stored on the computer readable medium for deriving a time series signal from each temporal, event-based segment in the set of temporal, event-based segments using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment; and
program code stored on the computer readable medium for extracting a segment fingerprint based on the time series signal for each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content.

### Claim 13 (depends on 12)

The computer program product of claim 12 wherein the set of temporal segment based fingerprints are used to determine if a portion of a first video clip is derived from a same original content as a second video clip, the computer program product further comprising:
program code stored on the computer readable medium for comparing temporal segment based fingerprints for the first video clip with temporal segment based fingerprints generated for the second video clip;
program code stored on the computer readable medium for identifying matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;
program code stored on the computer readable medium for collecting all matching segments between the first video clip and the second video clip to form a set of matching segments;
program code stored on the computer readable medium for selecting a subset of matching segments, wherein the subset of matching segments comprises matching segments associated with the first video clip that produces a good linear fit to matching segments associated with the second video clip;
program code stored on the computer readable medium for identifying an overall video match score for the first video clip and the second video clip based on the selected matching segments in the subset of matching segments; and
program code stored on the computer readable medium for determining whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

### Claim 14 (depends on 13)

The computer program product of claim 13 further comprising:
program code stored on the computer readable medium for comparing the overall video match score to a threshold score;
program code stored on the computer readable medium for identifying the first video clip as a near-duplicate of the second video clip in response to the overall video match score exceeding the threshold score.

### Claim 15 (depends on 12)

The computer program product of claim 12 wherein the content-based features comprises at least one of audio features and visual features, and wherein each event is detected based on at least one of video shot detection, scene changes, speaker changes, audio changes, frame intensity changes, or changes based on low-level content-based descriptors of color, texture, shape, edges, or motion.

### Claim 16

An apparatus for automatically detecting video piracy, the apparatus comprising:
a bus system;
a communications system coupled to the bus system;
a memory connected to the bus system, wherein the memory includes computer usable program code; and
a processing unit coupled to the bus system, wherein the processing unit executes the computer usable program code to detect events in video content, wherein the video content comprises a plurality of video frames, and wherein an event represents discrete points of interest in the video content; generate a set of temporal, event-based segments using the events, wherein each temporal, event-based segment is a segment of the video content covering a set of events; derive a time series signal from each temporal, event-based segment in the set of temporal, event-based segments using temporal tracking of content-based features of a set of frames associated with the each temporal, event-based segment; and extract a segment fingerprint based on the time series signal for each temporal, event-based segment to form a set of temporal segment based fingerprints associated with the video content, wherein detection of the near-duplicate is used for content-based video piracy detection.

### Claim 17 (depends on 16)

The apparatus of claim 16 wherein the processing unit further executes the computer usable program code to compare temporal segment based fingerprints for the first video clip with temporal segment based fingerprints generated for the second video clip; identify matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment; collect all matching segments between the first video clip and the second video clip to form a set of matching segments; select a subset of matching segments, wherein the subset of matching test segments produces a good linear fit to the matched video; identify an overall video match score for the first video clip and the second video clip based on the selected matching segments in the subset of matching segments; and determine whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

### Claim 18 (depends on 17)

The apparatus of claim 17 wherein the processing unit further executes the computer usable program code to compare the overall video match score to a threshold score; and identify the first video clip as a near-duplicate of the second video clip in response to the overall video match score exceeding the threshold score.

### Claim 19

A computer implemented method for automatically detecting video piracy, the computer implemented method comprising:
comparing temporal segment based fingerprints for a first video clip with temporal segment based fingerprints generated for a second video clip;
identifying matching event-based segments based on a similarity measure between a temporal segment-based fingerprint associated with the first video clip and a temporal segment based fingerprint associated with the second video clip to form a matching segment;
collecting all matching segments between the first video clip and the second video clip to form a set of matching segments;
selecting a subset of matching segments, wherein the subset of matching segments comprises matching segments associated with the first video clip that produces a good linear fit to matching segments associated with the second video clip;
identifying an overall video match score for the first video clip and the second video clip based on the selected matching segments in the subset of matching segments;
determining whether the first video clip is a near-duplicate of the second video clip using the overall video match score.

### Claim 20 (depends on 19)

The computer implemented method of claim 19 further comprising:
comparing the overall video match score to a threshold score; and
identifying the first video clip as a near-duplicate of the second video clip in response to the overall video match score exceeding the threshold score.
