# US 2024/0193478 A1 · Training data evaluation system, method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0193478 A1
- Document ID: US20240193478A1
- Status: Published application
- Kind: A1
- Application number: 18/446,909
- Filed: 2023-08-09
- Published: 2024-06-13
- Earliest claimed: 2022-12-08
- Assignees: Hitachi Solutions, Ltd.
- Applicants: Hitachi Solutions, Ltd.
- Inventors: Yuxin LIANG; Masashi EGI; Koji NAKAYAMA
- Web page: https://patentgrep.com/patent/US20240193478A1

## CPC

- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

It is possible to effectively and efficiently add training data. A training data evaluation system includes an uncertainty calculation unit configured to calculate, based on a prediction value obtained from a machine learning model using evaluation data for evaluating a shortage of training data as an input and a correct answer for the evaluation data, a data shortage degree representing a training data shortage degree for each piece of the evaluation data; a target selection unit configured to extract target data, which is data to be added to the training data, based on a predetermined selection rule and the data shortage degree; and a tendency analysis planning unit configured to specify a tendency of the target data based on a predetermined analysis rule and specify a property of the training data to be added based on the tendency of the target data.

## Claims

### Claim 1

A training data evaluation system comprising:
an uncertainty calculation unit configured to calculate, based on a prediction value obtained from a machine learning model using evaluation data for evaluating a shortage of training data as an input and a correct answer for the evaluation data, a data shortage degree representing a training data shortage degree for each piece of the evaluation data;
a target selection unit configured to extract target data, which is data to be added to the training data, based on a predetermined selection rule and the data shortage degree; and
a tendency analysis planning unit configured to specify a tendency of the target data based on a predetermined analysis rule and specify a property of the training data to be added based on the tendency of the target data.

### Claim 2 (depends on 1)

The training data evaluation system according to claim 1, wherein
the data shortage degree includes a fluctuation value representing a degree of fluctuation in the prediction value when weight in the machine learning model is modified,
the selection rule is predetermined including a rule for classifying the target data based on the fluctuation value, and
the target selection unit extracts the target data based on the selection rule and the fluctuation value.

### Claim 3 (depends on 2)

The training data evaluation system according to claim 2, wherein
the selection rule is predetermined including a rule for setting data, as the target data, in which the fluctuation value is in a predetermined ratio from an upper level.

### Claim 4 (depends on 3)

The training data evaluation system according to claim 3, wherein
the machine learning model is a model for detecting a predetermined object from an image, and
the selection rule includes a rule in which an IoU of a prediction value for the correct answer is smaller than a predetermined threshold value and the data in which the fluctuation value is in the ratio from the upper level is set as the target data.

### Claim 5 (depends on 2)

The training data evaluation system according to claim 2, wherein
the machine learning model is a model for identifying what appears in an image, and
the selection rule includes a rule for setting evaluation data, as the target data, in which the fluctuation value is greater than a predetermined threshold value.

### Claim 6 (depends on 1)

The training data evaluation system according to claim 1, wherein
the analysis rule is predetermined including a rule for classifying the target data into a plurality of classification groups according to a common property, and
the tendency analysis planning unit classifies the target data into the classification groups based on the analysis rule, and specifies a property of the training data to be added for each classification group.

### Claim 7 (depends on 6)

The training data evaluation system according to claim 6, wherein
the machine learning model is a model for detecting a predetermined object from an image,
the analysis rule is predetermined including a rule for classifying the target data in which a color variance of an object in the correct answer is greater than a predetermined threshold value into one classification group, and
the tendency analysis planning unit proposes that an image having the color variance greater than the threshold value is to be generated based on the target data classified into the classification group and is to be added to the training data.

### Claim 8 (depends on 6)

The training data evaluation system according to claim 6, wherein
the machine learning model is a model for identifying what appears in an image,
the analysis rule includes a rule for classifying the target data into a target classification group based on a distribution of pixels in an image in which a contribution to a pixel fluctuation value is greater than a predetermined threshold value, and
the tendency analysis planning unit proposes that an image having the distribution of the target classification group is to be generated from the target data classified into the target classification group based on the analysis rule and is to be added to the training data.

### Claim 9 (depends on 8)

The training data evaluation system according to claim 8, further comprising:
a display unit configured to display an improvement proposal in which a property common to the target classification group and an improvement plan to be added with the training data having the image based on the distribution of the target classification group are associated.

### Claim 10 (depends on 8)

The training data evaluation system according to claim 8, wherein
the contribution is a value calculated by the following equation.
Contribute
(
c
,
x
,
x
′
)
=
K
-
1
K
2
⁢
(
x
-
x
′
)
⁢
∑
k
=
1
K
(
IG
2
(
c
,
k
,
x
,
x
′
)
-
IG
⁡
(
c
,
k
,
x
,
x
′
)
*
<
IG
⁡
(
c
,
k
,
x
,
x
′
)
>
in which
x: input feature data
x′: reference feature data to be compared
K: the total number of weight modification patterns
M: the number of divisions in trapezoidal approximation of integral
pc,k (x): prediction of class c for input feature data x in a model with a weight modification pattern k
∇x pc,k (x): partial differentiation of pc,k (x) to pixel feature data x
pc,k2 (x): a predicted squared value of class c for the pixel feature data x in a weight modification pattern k model
∇x pc,k2 (x): partial differentiation of pc,k (x) to the pixel feature data x
IG
⁡
(
c
,
k
,
x
,
x
′
)
=
(
x
-
x
′
)
⁢
∫
0
1
∇
x
p
c
,
k
(
x
′
+
α
⁡
(
x
-
x
′
)
)
⁢
d
⁢
α
≈
(
x
-
x
′
)
⁢
∑
j
=
1
M
∇
x
p
c
,
k
(
x
′
+
j
M
⁢
(
x
-
x
′
)
)
M
⁢
IG
⁡
(
c
,
k
,
x
,
x
′
)
=
(
x
-
x
′
)
⁢
∫
0
1
∇
x
p
c
,
k
2
(
x
′
+
α
⁡
(
x
-
x
′
)
)
⁢
d
⁢
α
≈
(
x
-
x
′
)
⁢
∑
j
=
1
M
∇
x
p
c
,
k
2
(
x
′
+
j
M
⁢
(
x
-
x
′
)
)
M
<IG(c, k, x, x′)>: an average value of IG(c, k, x, x′) obtained by a model “other than” the weight modification pattern k For example, when there are three weight modification patterns k (k=1, k=2, and k=3),
<
IG
⁡
(
c
,
k
=
1
,
x
,
x
′
)
>=
IG
⁡
(
c
,
k
=
2
,
x
,
x
′
)
+
IG
⁡
(
c
,
k
=
3
,
x
,
x
′
)
2

### Claim 11 (depends on 8)

The training data evaluation system according to claim 8, wherein
the contribution is a value calculated by the following equation.
Contribute
(
c
,
x
,
x
′
)
=
(
x
-
x
′
)
⁢
∫
0
1
dv
⁡
(
(
x
′
+
α
⁡
(
x
-
x
′
)
)
)
dx
⁢
d
⁢
α
≈
(
x
-
x
′
)
⁢
1
K
⁢
∑
m
=
1
M
dv
⁡
(
x
′
+
m
M
⁢
(
x
-
x
′
)
)
dx
in which
x: input feature data
x′: reference feature data to be compared
K: the total number of weight modification patterns
M: the number of divisions in trapezoidal approximation of integral
pc,k (x): prediction of class c for input feature data x in a model with a weight modification pattern k
v(x): a variance in prediction of pc,k (x)
dv
⁡
(
x
)
dx
:
a partial differentiation value of v(x) to x
v
⁡
(
x
)
=
∑
k
=
1
K
p
c
,
k
⁢
(
x
)
2
-
1
K
⁢
(
∑
k
=
1
K
p
c
,
k
⁢
(
x
)
)
2
;
dv
⁡
(
x
)
dx
=
∑
k
=
1
K
2
⁢
p
c
,
k
⁢
(
x
)
⁢
dp
c
,
k
⁢
(
x
)
dx
-
(
∑
k
=
1
K
2
K
2
⁢
p
c
,
k
⁢
(
x
)
)
*
∑
k
=
1
K
dp
c
,
k
⁢
(
x
)
dx

### Claim 12 (depends on 1)

The training data evaluation system according to claim 1, wherein
the machine learning model is a model in which one or more rectangles that indicate an area that is estimated to be an area in which a predetermined object exists in an image are output as prediction value candidates,
the uncertainty calculation unit sets, as a prediction value for the evaluation data, a prediction value candidate of a rectangle having a highest similarity to a rectangle representing the correct answer to the evaluation data.

### Claim 13

A training data evaluation method using a device having a processing device, comprising:
an uncertainty calculation step of calculating, based on a prediction value obtained from a machine learning model using evaluation data for evaluating a shortage of training data as an input and a correct answer for the evaluation data, a data shortage degree representing a training data shortage degree for each piece of the evaluation data;
a target selection step of extracting target data, which is data to be added to the training data, based on a predetermined selection rule and the data shortage degree; and
a tendency analysis planning step of specifying a tendency of the target data based on a predetermined analysis rule and specifying a property of the training data to be added based on the tendency of the target data.

### Claim 14

A training data evaluation program that causes a device having a processing device to implement:
an uncertainty calculation function configured to calculate, based on a prediction value obtained from a machine learning model using evaluation data for evaluating a shortage of training data as an input and a correct answer for the evaluation data, a data shortage degree representing a training data shortage degree for each piece of the evaluation data;
a target selection function of extracting target data, which is data to be added to the training data, based on a predetermined selection rule and the data shortage degree; and
a tendency analysis planning function of specifying a tendency of the target data based on a predetermined analysis rule and specifying a property of the training data to be added based on the tendency of the target data.
