US 10,282,616 B2Granted

Visual data mining

AssigneeVision Semantics Limited

InventorsShaogang Sean Gong, Timothy Miguel Hospedales

Life of this patent

Estimating the term…

Loading patent…

Abstract

Method and system for finding targets within visual data, comprising receiving target object information. Generating a set of target object semantic attributes from the target object information. Identifying a plurality of candidate objects within visual data. Generating a set of low-level feature descriptors from the visual data for each candidate object. Generating from the set of low-level feature descriptors a set of candidate semantic attributes for each candidate object within the visual data. Identifying one or more portions of the visual data containing a candidate object, from the plurality of candidate objects, having a set of candidate object semantic attributes that match the set of target object semantic attributes. Providing an output indicating the identified one or more portions of the visual data.

Claims18

Dependent claims

A method for finding a target within visual data, the method comprising the steps of:

receiving target object information identifying one or more visual characteristics of the target;
generating a set of target object semantic attributes to represent the target, each semantic attribute associated with a visual characteristic identified in the target object information;
identifying a plurality of portions within visual data, each representing a candidate object being a visual object that is a potential visual match to the target;
for each portion of visual data representing a candidate object, generating a set of low-level feature descriptors from the visual data;
generating from the set of low-level feature descriptors a set of semantic attributes representing each candidate object, by identifying semantic attributes within a database associated with stored low-level feature descriptors that match the generated low-level feature descriptors;
identifying one or more of the plurality of candidate objects having greatest similarity to the target;
and
providing a filtered output indicating the portions of the visual data which the one or more identified candidate objects represents, wherein the filtered output corresponds to less than all the visual data,
wherein the step of identifying one or more of the plurality of candidate objects having greatest similarity to the target comprises:
computing a distance metric, d, between each candidate object and the target object according to:
d(xi,xj)=wLLFdLLF(xi,xj)+wA(A(xi)−A(xj))TM(A(xi)−A(xj))
where i and j index target and candidate images respectively;
wLLF and wA weight the contributions of low-level features and semantic attributes, respectively;
for an image x then A(x) is a set of sematic attributes represented by an attribute profile vector;
and M is a diagonal or full Mahalanobis matrix to encode weights for individual semantic attributes and semantic attribute correlations, respectively;
wherein a gradient-based optimisation is used to determine weights wLLF and wA, such that weight wLLF for each low level feature and weight wA for each semantic attribute are set by minimizing an objective function for a training probe set P and a gallery set G according to:
ER
⁡
(
w
LLF
,
w
A
,
M
)
=
1
P
⁢
∑
p
∈
P
⁢
∑
g
∈
G
⁢
σ
M
⁡
(
d
pp
-
d
pg
)
+
λ
⁢

w
-
w
0

2
where σM is a sigmoid function which makes the objective function smooth, dpp is the distance of true matches, dpg are the distances of false matches, and λ and w0 are an optional regularizer strength and bias.

59 more elements

A method of re-identifying one or more targets within visual data, the method comprising the steps of:

receiving target object information identifying one or more visual characteristics of the target;
generating a set of semantic attributes to represent the target, each semantic attribute associated with a visual characteristic identified in the target object information;
identifying a plurality of portions within visual data, each representing a candidate object being a visual object that is a potential visual match to the target;
for each portion of visual data representing a candidate object, generating a set of low-level feature descriptors from the visual data;
generating from the set of low-level feature descriptors a set of semantic attributes representing each candidate object, by identifying semantic attributes within a database associated with stored low-level feature descriptors that match the generated low-level feature descriptors;
identifying one or more of the plurality of candidate objects having greatest similarity to the target;
and
outputting an indication of filtered portions of the visual data containing the identified one or more of the plurality of candidate objects, wherein the filtered portions of the visual data are less than all the visual data,
wherein the step of identifying one or more of the plurality of candidate objects having greatest similarity to the target comprises:
computing a distance metric, d, between each candidate object and the target object according to:
d(xi,xj)=wLLFdLLF(xi,xj)+wA(A(xi)−A(xj))TM(A(xi)−A(xj))
where i and j index target and candidate images respectively;
wLLF and wA weight the contributions of low-level features and semantic attributes, respectively;
for an image x then A(x) is a set of sematic attributes represented by an attribute profile vector;
and M is a diagonal or full Mahalanobis matrix to encode weights for individual semantic attributes and semantic attribute correlations, respectively;
wherein a gradient-based optimisation is used to determine weights wLLF and wA, such that weight wLLF for each low level feature and weight wA for each semantic attribute are set by minimizing an objective function for a training probe set P and a gallery set G according to:
ER
⁡
(
w
LLF
,
w
A
,
M
)
=
1
P
⁢
∑
p
∈
P
⁢
∑
g
∈
G
⁢
σ
M
⁡
(
d
pp
-
d
pg
)
+
λ
⁢

w
-
w
0

2
where σM is a sigmoid function which makes the objective function smooth, dpp is the distance of true matches, dpg are the distances of false matches, and λ and w0 are an optional regularizer strength and bias.

59 more elements

A system for identifying targets within visual data, comprising:

an output device;
and
a processor configured to:
receive target object information identifying one or more visual characteristics of the target;
generate a set of semantic attributes to represent the target, each semantic attribute associated with a visual characteristic identified in the target object information;
identify a plurality of portions within visual data, each representing a candidate object being a visual object that is a potential visual match to the target;
for each portion of visual data representing a candidate object, generate a set of low-level feature descriptors from the visual data for each candidate object;
generate from the set of low-level feature descriptors a set of semantic attributes representing each candidate object, by identifying semantic attributes within a database associated with stored low-level feature descriptors that match the generated low-level feature descriptors;
identify one or more of the plurality of candidate objects having greatest similarity to the target;
and
send to the output device a filtered output indicating the portions of the visual data which the one or more identified candidate objects represents, wherein the filtered output corresponds to less than all the visual data,
wherein to identify one or more of the plurality of candidate objects having greatest similarity to the target, the processor is further configured to:
compute a distance metric, d, between each candidate object and the target object according to:
d(xi,xj)=wLLFdLLF(xi,xj)+wA(A(xi)−A(xj))TM(A(xi)−A(xj))
where i and j index target and candidate images respectively;
wLLF and wA weight the contributions of low-level features and semantic attributes, respectively;
for an image x then A(x) is a set of sematic attributes represented by an attribute profile vector;
and M is a diagonal or full Mahalanobis matrix to encode weights for individual semantic attributes and semantic attribute correlations, respectively;
wherein a gradient-based optimisation is used to determine weights wLLF and wA, such that weight wLLF for each low level feature and weight wA for each semantic attribute are set by minimizing an objective function for a training probe set P and a gallery set G according to:
ER
⁡
(
w
LLF
,
w
A
,
M
)
=
1
P
⁢
∑
p
∈
P
⁢
∑
g
∈
G
⁢
σ
M
⁡
(
d
pp
-
d
pg
)
+
λ
⁢

w
-
w
0

2
where σM is a sigmoid function which makes the objective function smooth, dpp is the distance of true matches, dpg are the distances of false matches, and λ and w0 are an optional regularizer strength and bias.

62 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/028,082Filed 17 Oct 2014
Priority: GB 1318472.618 Oct 2013
Published: US 2016/0239711 A118 Aug 2016
Granted: 7 May 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06V 40/173
Classification, e.g. identification: face re-identification, e.g. recognising unknown faces across different face tracks
G06F 16/5838
Using metadata automatically derived from the content: using colour
G06F 16/735
Querying: filtering based on additional data, e.g. user or group profiles
G06V 20/41
In video content: higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian…