# US 10,373,320 B2 · Method for detecting moving objects in a video having non-stationary background

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,320 B2
- Document ID: US10373320B2
- Status: Granted
- Kind: B2
- Application number: 15/462,530
- Filed: 2017-03-17
- Published: 2018-09-20
- Granted: 2019-08-06
- Earliest claimed: 2017-03-17
- Assignees: Uurmi Systems Pvt Ltd
- Applicants: Uurmi Systems Pvt Ltd
- Inventors: Adithya Apuroop Karavadi; Ajinkya Santoshrao Deshmukh; Shanti Swarup Medasani
- Web page: https://patentgrep.com/patent/US10373320B2

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30241: Trajectory
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Method and system for tracking moving objects in a video with non-stationary background is provided. The method comprises estimation of non-stationary background and determining an approximate foreground for each image that learns the background model over time by incorporating various constraints on image pixels. Then, weights based upon spatio-temporal statistical properties of the extracted foreground blobs and blob edge overlap are used to identify and track with bounding boxes displayed for one or more true objects.

## Claims

### Claim 1

A method for detecting one or more moving objects in a video comprising a plurality of images, each image having a foreground and a background, the method comprising the steps of:
determining an approximate foreground for each image by
determining background segment of each image and determining image segment of each image and computing an element by element product, the background segment is determined based on
S
B
⁡
(
i
,
j
)
=
{
1
,
if
⁢
|
H
B
ij
⁡
(
b
⁡
(
x
)
)
|
<
a
1
+
ϵ
⁢
⁢
h
B
ij
0
,
else
where HBij(b(x)) denotes the bins corresponding to the background intensity at location (i,j), hB=maxk HB(k) denotes the values of maximum number of occurrences of a bin, a1 is a predefined constant and ϵhBij is a parameter added to account for noise and compression artefacts, and the image segment is determined based on
S
I
⁡
(
i
,
j
)
=
{
1
,
if
⁢
|
H
I
ij
⁡
(
b
⁡
(
x
)
)
|
<
c
⁡
(
i
,
j
)
⁢
h
I
ij
0
,
else
where HIij(b(x)) denotes the bins corresponding to the original pixel intensity at location (i,j), hl=maxk Hl(k) and denotes the values of maximum number of occurrences of a bin and c=0.5(1−2σ) is the pixel level threshold with σ being the standard deviation in the background;
refining the segments of each of the plurality of images by setting pixels in the segment of each of the plurality of images to zero if the corresponding image segment differs from mean background by less than μ+σ where μ is the minimum intensity difference for two pixels to be considered dissimilar; and
relabeling the foreground samples as background that intersect with more than a1 samples in the background model;
updating the most recent background frame B as B=B⊙S+I⊙(1−S) using the segmentation decision where S is foreground and I is image;
extracting all the foreground blobs having size greater than a predefined threshold;
assigning statistical weights for size/area of the extracted foreground blobs, intensity, distance, displacement vectors, and dilated edges measures across frames for the extracted one or more blobs and searching for correspondences for the one or more blobs in the current frame by calculating a final weight as a combination of an initial weight and a similarity weight with a pre-determined learning rate, wherein the assigning of the statistical weights for size by calculating the similarity weight with respect to size comprises:
imposing a penalty on randomly varying blob sizes from frame to frame; and
calculating the similarity size weight Wsim as an exponential function of several parameters including a, a2, anextj, and βj, wherein a denotes area of a frame pk(i), a2 denotes a predefined constant for allowing foreground area size fluctuations in blob analysis to account for small sized objects, anextj denotes area of a next frame Nj(pk(i)), and βj denotes a predefined parameter;
identifying potential targets by assigning a weight to the map from one or more blobs in the current frame to the one or more potential correspondences in the next frame, wherein identifying includes selecting the blob with the highest total positive weight as the blob corresponding to the one or more blobs in the current frame; and
tracking potential targets and displaying the bounding box for one or more true objects, by selecting one or more potential blobs whose weights of the mapping exceeds a predetermined threshold for more than predefined minimum frames continuously.

### Claim 2 (depends on 1)

The method of claim 1, wherein the objects are detected independent of size and velocity.

### Claim 3 (depends on 1)

The method of claim 1, wherein the image comprises a dynamic movement in the background.

### Claim 4 (depends on 1)

The method of claim 1, wherein the dynamic movement comprises one or more of snow, trees waving, rain, or atmospheric turbulence.

### Claim 5 (depends on 1)

The method of claim 1, further comprising using a tracking algorithm after detecting one or more true objects in a current frame for updating and predicting undetected objects in one or more next frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein the initial weight is calculated by:
initializing one or more blobs in the current frame with weight of the corresponding mapping that have a corresponding blob in the previous frame and with a negative weight to those that have no previous correspondences, wherein the weight depends on the overlap of the blobs with the edges; and
calculating the initial weight Winitial(pk(i)) for the blobs pk(i) with the current frame index k as
W
initial
⁡
(
p
k
⁡
(
i
)
)
=
{
W
⁡
(
⁢
(
p
k
⁡
(
i
)
)
→
p
k
⁡
(
i
)
)
,
if
⁢
⁢
W
⁡
(
⁢
(
p
k
⁡
(
i
)
)
→
p
k
⁡
(
i
)
)
>
0
-
*
|
p
k
⁡
(
i
)
⋂
E
D
|
,
else
,
where is a penalty parameter, ED is a set of dilated edge pixels and (pk(i)) is the blob in the previous frame corresponding to pk(i), i=1, 2, . . . , wherein W(x) indicates the weight given to the blob x and W(x→y) denotes the weight of the mapping of blob x to blob y.

### Claim 7 (depends on 1)

The method of claim 1, wherein the similarity size weight Wsim is calculated as
W
sim
⁡
(
N
j
⁡
(
p
k
⁡
(
i
)
)
)
=
-
1
+
2
⁢
⁢
exp
⁡
(
-
β
j
|
log
⁡
(
a
2
+
a
next
j
a
2
+
a
)
|
)
,
where a=area(pk(i)) and
a
next
j
=
area
⁡
(
N
j
⁡
(
p
k
⁡
(
i
)
)
)
,
β
j
=
1
+
β
0
⁢
⁢
d
next
j
M
2
+
N
2
indicates the predefined parameter, β0 is a positive constant and dnextj is the distance between the current blob and its jth neighbor in the next frame Nj(pk(i)).

### Claim 8 (depends on 1)

The method of claim 1, wherein the step of assigning statistical weight for intensity by calculating the similarity weight with respect to intensity further comprising the steps of:
calculating the similarity intensity weight as Wsim(Nj(pk(i)))=Wsim(Nj(pk(i)))−1+2 exp(−βj*(cnextj−c)2), where c=I(pk(i)) is intensity for current blob and cnextj=I(Nj(pk(i))) denotes intensity for neighboring blob in next frame, I(Nj(pk(i))) denotes the average over the blob pixels, and is a predefined constant.

### Claim 9 (depends on 1)

The method of claim 1, wherein the step of assigning statistical weight for distance by calculating the similarity weight with respect to distance further comprising the steps of:
capturing the motion of true objects by including a trust parameter t and using it to calculate the similarity distance weight, wherein the trust parameter is t(pk(i))=t0(1+t*(pk(i))), where t0 and t are predefined positive constants and (pk(i)) denotes the number of previous frames for which pk(i) has correspondences; and
calculating the similarity distance weight as
W
sim
⁡
(
N
j
⁡
(
p
k
⁡
(
i
)
)
)
=
W
sim
⁡
(
N
j
⁡
(
p
k
⁡
(
i
)
)
)
+
or
⁢
⁢
t
⁡
(
p
k
⁡
(
i
)
)
*
(
-
1
+
2
⁢
⁢
exp
⁡
(
-
β
j
|
log
⁡
(
d
1
+
d
next
j
d
1
+
d
)
|
)
)
,
where d denotes the distance traveled by the blob from the previous frame to the current frame and dnextj denotes the distance between the current blob and its jth neighbour in the next frame, and d1 is a predefined constant for allowing distance fluctuations.

### Claim 10 (depends on 1)

The method of claim 1, wherein the step of assigning statistical weight for displacement vectors by calculating the similarity weight with respect to displacement vectors further comprising the steps of:
using the cosine of the angle θ between the displacement vectors k(pk(i))−initial(pk(i)) and k+1(N(pk(i))−initial(pk(i)) as a measure of consistency in velocity, where initial(pk(i)) denote the coordinates of the blob corresponding to pk(i) when the correspondences first began, and k(pk(i)) denotes the coordinates of the ith blob in the current frame; and
calculating the similarity displacement weight as Wsim(Nj(pk(i)))=Wsim(Nj(pk(i)))+cos(θ).

### Claim 11 (depends on 1)

The method of claim 1, wherein the step of assigning statistical weight for dilated edges by calculating the similarity weight with respect to dilated edges further comprising the steps of:
using a canny edge detector and dilating the resulting edges using a disk of a predefined radius and obtaining ED as the resulting set of dilated edge pixels; and
calculating the similarity weight for dilated edges as Wsim(Nj(pk(i)))=Wsim(Nj(pk(i)))+2|Nj(pk(i)∩ED|2.

### Claim 12 (depends on 1)

The method of claim 1, wherein the step of identifying potential targets further comprising the steps of:
assigning a weight to the map from one or more blobs in the current frame to the one or more potential correspondences in the next frame, wherein the weight is
W
⁡
(
p
k
⁡
(
i
)
→
N
j
⁢
⁢
(
p
k
⁡
(
i
)
)
)
=
exp
⁡
(
-
d
next
j
M
2
+
N
2
)
⁡
[
(
1
-
α
)
⁢
W
initial
⁡
(
p
k
⁡
(
i
)
)
+
α
⁢
⁢
W
sim
⁡
(
N
j
⁢
⁢
(
p
k
⁡
(
i
)
)
)
]
,
where α is a predefined constant learning rate and adding an exponential factor for further penalizing the distant blobs.

### Claim 13

A system for detecting one or more moving objects in a video comprising a plurality of images, each image having a foreground and a background, comprising:
a processor configured for executing one or more programming modules stored in a non-transient memory, wherein the memory comprises of:
a background estimation module that upon being executed by the processor determines
an approximate foreground for each image by determining background segment of each image and determining image segment of each image and computing an element by element product, the background segment is determined based on
S
B
⁡
(
i
,
j
)
=
{
1
,
if
⁢
|
H
B
ij
⁡
(
b
⁡
(
x
)
)
|
<
a
1
+
ϵ
⁢
⁢
h
B
ij
0
,
else
where HBij(b(x)) denotes the bins corresponding to the background intensity at location (i,j), hB=maxk HB(k) denotes the values of maximum number of occurrences of a bin, a1 is a predefined constant, and ϵhBij is a parameter added to account for noise and compression artefacts, and the image segment is determined based on
S
I
⁡
(
i
,
j
)
=
{
1
,
if
⁢
|
H
I
ij
⁡
(
b
⁡
(
x
)
)
|
<
c
⁡
(
i
,
j
)
⁢
h
I
ij
0
,
else
where HIij(b(x)) denotes the bins corresponding to the original pixel intensity at location (i,j), hl=maxk Hl(k) and denotes the values of maximum number of occurrences of a bin and c=0.5(1−2σ) is the pixel level threshold with σ being the standard deviation in the background;
refining the segments of each of the plurality of images by setting pixels in the segment of each of the plurality of images to zero if the corresponding image segment differs from mean background by less than μ+σ where μ is the minimum intensity difference for two pixels to be considered dissimilar;
relabeling the foreground samples as background that intersect with more than a1 samples in the background model; and
updating the most recent background frame B as B=B⊙S+I⊙(1−S) using the segmentation decision where S is foreground and I is image;
a foreground preprocessing module that upon being executed by the processor extracts all the foreground blobs having size greater than a predefined threshold;
a statistical weight assignment module that upon being executed by the processor identifies potential targets by assigning a weight to the map from one or more blobs in the current frame to the one or more potential correspondences in the next frame and by assigning statistical weights for size/area of the extracted foreground blobs, wherein identifying includes selecting the blob with the highest total positive weight as the blob corresponding to the one or more blobs in the current frame, and wherein the assigning of the statistical weights for size by calculating a similarity weight with respect to size comprises:
imposing a penalty on randomly varying blob sizes from frame to frame; and
calculating the similarity size weight Wsim as an exponential function of several parameters including a, a2, anextj, and βj, wherein a denotes area of a frame pk(i), a2 denotes a predefined constant for allowing foreground area size fluctuations in blob analysis to account for small sized objects, anextj denotes area of a next frame Nj(pk(i)), and βj denotes a predefined parameter;
a potential targets identification module that upon being executed by the processor identifying potential targets by assigning a weight to the map from one or more blobs in the current frame to the one or more potential correspondences in the next frame, wherein identifying includes selecting the blob with the highest total positive weight as the blob corresponding to the one or more blobs in the current frame; and
a tracking objects detection module that upon being executed by the processor tracks potential targets and displaying the bounding box for one or more true objects, wherein detection of one or more true objects is performed by selecting one or more potential blobs whose weights of the mapping exceeds a predetermined threshold for more than predefined minimum frames continuously.

### Claim 14 (depends on 13)

The system of claim 13, wherein the system accounts for small camera jitters.

### Claim 15

A non-transient computer-readable medium, including instructions stored thereon which, when read and executed by one or more computers, causes the one or more computers to perform the method comprising the steps of:
determining an approximate foreground for each image by
determining background segment of each image and determining image segment of each image and computing an element by element product, the background segment is determined based on
S
B
⁡
(
i
,
j
)
=
{
1
,
if
⁢
|
H
B
ij
⁡
(
b
⁡
(
x
)
)
|
<
a
1
+
ϵ
⁢
⁢
h
B
ij
0
,
else
where HBij(b(x)) denotes the bins corresponding to the background intensity at location (i,j), hB=maxk HB(k) denotes the values of maximum number of occurrences of a bin, a1 is a predefined constant, and ϵhBij is a parameter added to account for noise and compression artefacts, and the image segment is determined based on
S
I
⁡
(
i
,
j
)
=
{
1
,
if
⁢
|
H
I
ij
⁡
(
b
⁡
(
x
)
)
|
<
c
⁡
(
i
,
j
)
⁢
h
I
ij
0
,
else
where HIij(b(x)) denotes the bins corresponding to the original pixel intensity at location (i,j), hl=maxk Hl(k) and denotes the value of maximum number of occurrences of a bin and c=0.5(1−2σ) is the pixel level threshold with σ being the standard deviation in the background; and
refining the segments of each of the plurality of images by setting pixels in the segment of each of the plurality of images to zero if the corresponding image segment differs from mean background by less than μ+σ where μ is the minimum intensity difference for two pixels to be considered dissimilar;
relabeling the foreground samples as background that intersect with more than a1 samples in the background model; and
updating the most recent background frame B as B=B⊙S+I⊙(1−S) using the segmentation decision where S is foreground and I is image;
extracting all the foreground blobs having size greater than a predefined threshold;
assigning statistical weights for size/area of the extracted foreground blobs, intensity, distance, displacement vectors, and dilated edges measures across frames for the extracted one or more blobs and searching for correspondences for the one or more blobs in the current frame by calculating a final weight as a combination of an initial weight and a similarity weight with a pre-determined learning rate, wherein the assigning of the statistical weights for size by calculating a similarity weight with respect to size comprises:
imposing a penalty on randomly varying blob sizes from frame to frame; and
calculating the similarity size weight Wsim as an exponential function of several parameters including a, a2, anextj, and βj, wherein α denotes area of a frame pk(i), a2 denotes a predefined constant for allowing foreground area size fluctuations in blob analysis to account for small sized objects, anextj denotes area of a next frame Nj(pk(i)), and βj denotes a predefined parameter;
identifying potential targets by assigning a weight to the map from one or more blobs in the current frame to the one or more potential correspondences in the next frame, wherein identifying includes selecting the blob with the highest total positive weight as the blob corresponding to the one or more blobs in the current frame; and
tracking potential targets and displaying the bounding box for one or more true objects, by selecting one or more potential blobs whose weights of the mapping exceeds a predetermined threshold for more than predefined minimum frames continuously.