# US 2021/0241140 A1 · Hybrid methods and systems for feature selection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241140 A1
- Document ID: US20210241140A1
- Status: Published application
- Kind: A1
- Application number: 16/782,730
- Filed: 2020-02-05
- Published: 2021-08-05
- Earliest claimed: 2020-02-05
- Assignees: The Florida International University Board of Trustees
- Inventors: Thejas Gubbi SADASHIVA; Sundararaj S. IYENGAR
- Web page: https://patentgrep.com/patent/US20210241140A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/285: {Clustering or classification}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems for feature selection (FS) in machine learning (ML) are provided. The filter and wrapper methods can be combined to provide a hybrid FS method and system. The data can be clustered using mini-batch K-means clustering and ranked using normalized mutual information (NMI). The wrapper method can include using either a feature inclusion process or a least-ranked feature exclusion process that eliminates least-ranked features one by one from the ranking list.

## Claims

### Claim 1

A system for performing feature selection in machine learning, the system comprising:
a processor; and
a machine-readable medium in operable communication with the processor and comprising instructions stored thereon that, when executed by the processor, perform the following steps:
receiving a dataset;
performing feature ranking on the dataset using a filter technique to obtain a ranking list of features; and
performing feature selection on the ranking list using a wrapper technique,
the filter technique comprising clustering data of the dataset and then using normalized mutual information (NMI) as a metric for ranking to generate the ranking list of features,
the NMI calculated as follows:
NMI
⁡
(
Ω
,
S
)
=
MI
⁡
(
Ω
;
S
)
[
G
⁡
(
Ω
)
+
G
⁡
(
S
)
)
]
/
2
,
where
MI
⁡
(
Ω
;
S
)
=
∑
k
⁢
∑
j
⁢
P
⁡
(
d
k
⋂
s
j
)
⁢
log
⁢
P
⁡
(
d
k
⋂
s
j
)
P
⁡
(
d
k
)
⁢
P
⁡
(
s
j
)
,
⁢
G
⁡
(
Ω
)
=
-
∑
k
⁢
(
d
k
)
⁢
log
⁢
⁢
P
⁡
(
d
k
)
,
and
where Ω is a set of clusters, S is a set of classes, P(dk)=probability of data in cluster dk, P(sj)=probability of data in cluster sj, G(S) is an entropy of the set of classes, and P(dk∩sj)=probability of data being in a convergence of dk and sj,
the use of the filter technique and the wrapper technique improving a runtime of the processor, and
the wrapper technique comprising performing a feature inclusion process.

### Claim 2 (depends on 1)

The system according to claim 1, the filter technique comprising K-means clustering.

### Claim 3 (depends on 1)

The system according to claim 1, the filter technique comprising mini-batch K-means clustering.

### Claim 4

(canceled)

### Claim 5 (depends on 1)

The system according to claim 1, the clustering of the data of the dataset comprising K-means clustering.

### Claim 6 (depends on 1)

The system according to claim 1, the clustering of the data of the dataset comprising mini-batch K-means clustering.

### Claim 7 (depends on 1)

The system according to claim 1, the wrapper technique comprising removing redundant features that have a dependency on other features.

### Claim 8

(canceled)

### Claim 9 (depends on 1)

The system according to claim 1, the feature inclusion process comprising performing Algorithm 1:
feature S = {f0, f1, f2, . . . fm}, where m = total number of features, obtained from the
feature ranking phase, f0 is the highest ranked feature and fm is the least ranked feature
Algorithm 1:
input: set of ranked features S = {f0, f1, f2, . . . fm}, where m = total number of features,
obtained from the feature ranking phase, f0 is the highest ranked feature and fm is the
least ranked feature
output: provides the selected set of features
initialization :
1:
Lst = S[0] prev = 0, where prev represents a previous accuracy of a model
2:
for k = 0 to m−1 do
3:
x_tst = x_tst [ Lst ]
4:
x_tr = x_tr [ Lst ]
5:
train the model based on any classifier and store an accuracy on acc
6:
if acc > prev then
7:
if (k ≠ m − 1) then
8:
add S[ k + 1 ] into the Lst
9:
prev = acc
10:
else
11:
end if
12:
else
13:
remove S [ k ] object from the Lst
14:
if (k ≠ m − 1) then
15:
add S[ k + 1 ] to the Lst
16:
else
17:
end if
18:
end if
19:
end for

### Claim at position 10

10-11. (canceled)

### Claim 12

A method for performing feature selection in machine learning, the method comprising:
receiving, by a processor, a dataset;
performing, by the processor, feature ranking on the dataset using a filter technique to obtain a ranking list of features; and
performing, by the processor, feature selection on the ranking list using a wrapper technique,
the filter technique comprising clustering data of the dataset and then using normalized mutual information (NMI) as a metric for ranking to generate the ranking list of features,
the NMI calculated as follows:
NMI
⁡
(
Ω
,
S
)
=
MI
⁡
(
Ω
;
S
)
[
G
⁡
(
Ω
)
+
G
⁡
(
S
)
)
]
/
2
,
where
MI
⁡
(
Ω
;
S
)
=
∑
k
⁢
∑
j
⁢
P
⁡
(
d
k
⋂
s
j
)
⁢
log
⁢
P
⁡
(
d
k
⋂
s
j
)
P
⁡
(
d
k
)
⁢
P
⁡
(
s
j
)
,
⁢
G
⁡
(
Ω
)
=
-
∑
k
⁢
(
d
k
)
⁢
log
⁢
⁢
P
⁡
(
d
k
)
,
and
where Ω is a set of clusters, S is a set of classes, P(dk)=probability of data in cluster dk, P(sj)=probability of data in cluster sj, G(S) is an entropy of the set of classes, and P(dk∩sj)=probability of data being in a convergence of dk and sj,
the use of the filter technique and the wrapper technique improving a runtime of the processor, and
the wrapper technique comprising performing a feature inclusion process.

### Claim 13 (depends on 12)

The method according to claim 12, the filter technique comprising mini-batch K-means clustering.

### Claim 14

(canceled)

### Claim 15 (depends on 12)

The method according to claim 12, the wrapper technique comprising removing redundant features that have a dependency on other features.

### Claim 16

(canceled)

### Claim 17 (depends on 12)

The method according to claim 12, the feature inclusion process comprising performing Algorithm 1:
Algorithm 1:
input: set of ranked features S = {f0, f1, f2, . . . fm}, where m = total number of features,
obtained from the feature ranking phase, f0 is the highest ranked feature and fm is the
least ranked feature
output: provides the selected set of features
initialization :
1:
Lst = S[0] prev = 0, where prev represents a previous accuracy of a model
2:
for k = 0 to m−1 do
3:
x_tst = x_tst [ Lst ]
4:
x_tr = x_tr [ Lst ]
5:
train the model based on any classifier and store an accuracy on acc
6:
if acc > prev then
7:
if (k ≠ m − 1) then
8:
add S[ k + 1 ] into the Lst
9:
prev = acc
10:
else
11:
end if
12:
else
13:
remove S [ k ] object from the Lst
14:
if (k ≠ m − 1) then
15:
add S[ k + 1 ] to the Lst
16:
else
17:
end if
18:
end if
19:
end for.

### Claim at position 17

18-19. (canceled)

### Claim 20

A system for performing feature selection in machine learning, the system comprising:
a processor; and
a machine-readable medium in operable communication with the processor and comprising instructions stored thereon that, when executed by the processor, perform the following steps:
receiving a dataset;
performing feature ranking on the dataset using a filter technique to obtain a ranking list of features; and
performing feature selection on the ranking list using a wrapper technique,
the filter technique comprising clustering data of the dataset and then using normalized mutual information (NMI) as a metric for ranking to generate the ranking list of features,
the NMI calculated as follows:
NMI
⁡
(
Ω
,
S
)
=
MI
⁡
(
Ω
;
S
)
[
G
⁡
(
Ω
)
+
G
⁡
(
S
)
)
]
/
2
,
where
MI
⁡
(
Ω
;
S
)
=
∑
k
⁢
∑
j
⁢
P
⁡
(
d
k
⋂
s
j
)
⁢
log
⁢
P
⁡
(
d
k
⋂
s
j
)
P
⁡
(
d
k
)
⁢
P
⁡
(
s
j
)
,
⁢
G
⁡
(
Ω
)
=
-
∑
k
⁢
(
d
k
)
⁢
log
⁢
⁢
P
⁡
(
d
k
)
,
where Ω is a set of clusters, S is a set of classes, P(dk)=probability of data in cluster dk, P(sj)=probability of data in cluster sj, G(S) is an entropy of the set of classes, and P(dk∩sj)=probability of data being in a convergence of dk and sj,
the clustering of the data of the dataset comprising mini-batch K-means clustering,
the wrapper technique comprising removing redundant features that have a dependency on other features,
the wrapper technique comprising performing a feature inclusion process,
the use of the filter technique and the wrapper technique improving a runtime of the processor, and
the feature inclusion process comprising performing Algorithm 1:
Algorithm 1:
input: set of ranked features S = {f0, f1, f2, . . . fm}, where m = total number of features,
obtained from the feature ranking phase, f0 is the highest ranked feature and fm is the
least ranked feature
output: provides the selected set of features
initialization :
1:
Lst = S[0] prev = 0, where prev represents a previous accuracy of a model
2:
for k = 0 to m−1 do
3:
x_tst = x_tst [ Lst ]
4:
x_tr = x_tr [ Lst ]
5:
train the model based on any classifier and store an accuracy on acc
6:
if acc > prev then
7:
if (k ≠ m − 1) then
8:
add S[ k + 1 ] into the Lst
9:
prev = acc
10:
else
11:
end if
12:
else
13:
remove S [ k ] object from the Lst
14:
if (k ≠ m − 1) then
15:
add S[ k + 1 ] to the Lst
16:
else
17:
end if
18:
end if
19:
end for.