US 8,027,542 B2Granted

High speed video action recognition and localization

AssigneeThe Regents of the University of California

InventorsParvez Ahammad, Chuohao Yeo, Kannan Ramchandran, S. Shankar Sastry

Life of this patent

Estimating the term…

Loading patent…

Abstract

An apparatus for detecting an action in a test video. In an illustrative embodiment, the apparatus includes a first mechanism for receiving a query for a particular action via a query video. A second mechanism employs motion vectors associated with the test video to compute one or more motion-similarity values. The one or more motion-similarity values represent motion similarity between a first group of pixels in a first frame of a query video and a second group of pixels in a second frame of the test video based on the motion vectors. A third mechanism uses the one or more similarity values to search for the particular action or similar action in the test video. In a more specific embodiment, another mechanism aggregates the similarity values over a predetermined number of frames to facilitate estimating where the particular action or version thereof occurs or is likely to occur in the test video.

Claims30

Dependent claims

A system comprising:

first means for receiving a query for a particular action, wherein the query includes a query video;
second means for computing motion similarity between a first group of pixels in a first frame of the query video and a second group of pixels in a second frame of a video via one or more motion vectors in the query video and the test video;
third means for providing one or more similarity values in response thereto;
forth means for aggregating the one or more similarity values, yielding one or more aggregated similarity values in response thereto;
fifth means for using the one or more similarity values to estimate where the particular action or version thereof occurs in the test video;
sixth means for estimating optical flows associated with the one or more motion vectors;
and
seventh means for computing a confidence score for the particular action based on the aggregated similarity values, wherein the confidence score (C(i)) is computed in accordance with the following equation:
C
⁡
(
i
)
=
max
max
⁡
(
i
-
T
/
2
,
1
)
≤
k
≤
(
i
+
T
/
2
,
Ttest
)
,
1
≤
j
≤
Tquery
⁢
S
a
⁡
(
k
,
j
)
,
wherein Tquery represents a number of frames in the query video;
Ttest represents a number of frames in the test video or portion thereof;
k is a frame counter corresponding to a given test frame;
i is a counter corresponding to a given test frame;
j is a counter corresponding to a given query frame;
C(i) is a confidence score for a T-length space-time (ST) region centered at frame i of the test video;
and Sa(k,j) is an aggregated similarity value estimating how similar a T-length video centered at frame k of the test video is to a T-length video centered at frame j of the query video.

58 more elements

A method comprising:

receiving a query for a particular action, wherein the query includes a query video;
computing motion similarity between a first group of pixels in a first frame of the query video and a second group of pixels in a second frame of a test video via one or more motion vectors in the query video and the test video;
providing one or more similarity values in response thereto;
aggregating the one or more similarity values, yielding one or more aggregated similarity values in response thereto;
using the one or more similarity values to estimate where the particular action or version thereof occurs in the test video;
estimating optical flows associated with the one or more motion vectors;
and
computing a confidence score for the particular action based on the aggregated similarity values, wherein the confidence score (C(i)) is computed in accordance with the following equation:
C
⁡
(
i
)
=
max
max
⁡
(
i
-
T
/
2
,
1
)
≤
k
≤
(
i
+
T
/
2
,
Ttest
)
,
1
≤
j
≤
Tquery
⁢
S
a
⁡
(
k
,
j
)
,
wherein Tquery represents a number of frames in the query video;
Ttest represents a number of frames in the test video or portion thereof;
k is a frame counter corresponding to a given test frame;
i is a counter corresponding to a given test frame;
j is a counter corresponding to a given query frame;
C(i) is a confidence score for a T-length space-time (ST) region centered at frame i of the test video;
and Sa(k,j) is an aggregated similarity value estimating how similar a T-length video centered at frame k of the test video is to a T-length video centered at frame j of the query video.

58 more elements

An apparatus comprising:

one or more processors;
and
a non-transitory computer-readable medium containing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method comprising:
receiving a query for a particular action, wherein the query includes a query video;
computing motion similarity between a first group of pixels in a first frame of the query video and a second group of pixels in a second frame of a test video via one or more motion vectors in the query video and the test video;
providing one or more similarity values in response thereto;
aggregating the one or more similarity values, yielding one or more aggregated similarity values in response thereto;
using the one or more similarity values to estimate where the particular action or version thereof occurs in the test video;
estimating optical flows associated with the one or more motion vectors;
and
computing a confidence score for the particular action based on the aggregated similarity values, wherein the confidence score (C(i)) is computed in accordance with the following equation:
C
⁡
(
i
)
=
max
max
⁡
(
i
-
T
/
2
,
1
)
≤
k
≤
(
i
+
T
/
2
,
Ttest
)
,
1
≤
j
≤
Tquery
⁢
S
a
⁡
(
k
,
j
)
,
where Tquery represents a number of frames in the query video;
Ttest represents a number of frames in the test video or portion thereof;
k is a frame counter corresponding to a given test frame;
i is a counter corresponding to a given test frame;
j is a counter corresponding to a given query frame;
C(i) is a confidence score for a T-length space-time (ST) region centered at frame i of the test video;
and Sa(k,j) is an aggregated similarity value estimating how similar a T-length video centered at frame k of the test video is to a T-length video centered at frame j of the query video.

61 more elements

Dependencies are inferred from the claim text.

Facts

Application: 11/764,754Filed 18 Jun 2007
Published: US 2008/0310734 A118 Dec 2008
Granted: 27 Sep 2011
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06V 20/40
Scenes; Scene-specific elements: in video content
G06F 16/70
Information retrieval; Database structures therefor; File system structures therefor: of video data
G08B 13/19613
Image analysis to detect motion of the intruder, e.g. by frame subtraction: recognition of a predetermined image pattern or behaviour pattern…
G06V
Image and video recognition