# US 12,118,714 B2 · Method of detecting and classifying defects and electronic device using the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,714 B2
- Document ID: US12118714B2
- Status: Granted
- Kind: B2
- Application number: 17/566,159
- Filed: 2021-12-30
- Published: 2022-06-30
- Granted: 2024-10-15
- Earliest claimed: 2020-12-31
- Assignees: Hon Hai Precision Industry Co., Ltd.
- Applicants: Hon Hai Precision Industry Co., Ltd.
- Inventors: Tung-Tso Tsai; Tzu-Chen Lin; Chin-Pin Kuo; Shih-Chao Chien
- Web page: https://patentgrep.com/patent/US12118714B2

## CPC

- G06T 7/001: {using an image reference approach}
- G06T 2207/20081: Training; Learning
- G06T 2207/30108: Industrial image inspection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method applied in an electronic device for detecting and classifying apparent defects in images of products inputs an image to a trained autoencoder to obtain a reconstructed image, determines whether the image reveals defects based on a defect criterion for filtering out small noise reconstruction errors. If so revealed, the electronic device calculates a plurality of structural similarity values between the image and a plurality of template images with marked defect categories, determines a target defect category corresponding to the highest structural similarity value, and classifies the defect revealed in the image into the target defect category.

## Claims

### Claim 1

A method for detecting and classifying defects comprising:
inputting an image to be detected to a trained autoencoder, and obtaining a reconstructed image corresponding to the image to be detected;
determining whether the image to be detected has defects based on a defect criteria for filtering out small noise reconstruction errors;
in response that the image to be detected has defects, calculating a plurality of structural similarity values between the image to be detected and a plurality of template images marking defect categories respectively;
determining a target defect category corresponding to the template image with the highest structural similarity value, and classifying the image to be detected into the target defect category, wherein a mathematical expression of the defect criteria is
∑
i
,
j
⁢
Δ
⁢
⁢
X
i
,
j
*
δ
⁢
⁢
X
i
,
j
∑
i
,
j
⁢
δ
⁢
⁢
X
i
,
j
>
τ
,
wherein ΔXi,j is a reconstruction error image between the image to be detected and the reconstructed image, δXi,j is a binary image of the reconstruction error image, i and j represent pixel positions, τ is a preset reconstruction error measurement threshold.

### Claim 2 (depends on 1)

The method for detecting and classifying defects as recited in claim 1, wherein τ is a statistical value of a reconstruction error corresponding to a reconstructed error image between the image and the reconstructed image, and the τ is adjusted based on a recall rate of a preset defect detection and an accuracy rate of the preset defect detection, the accuracy rate is a number of the images correctly detected as defective as a proportion of a number of all images detected as defective, and the recall rate is the number of the images correctly detected as defective as a proportion of the number of all images showing real defects.

### Claim 3 (depends on 1)

The method for detecting and classifying defects as recited in claim 1, wherein a binary image calculation formula of the binary image is
δ
⁢
⁢
X
i
,
j
=
{
1
Δ
⁢
⁢
X
i
,
j
>
ɛ
0
otherwise
,
ε is a filtering threshold of small noise reconstruction error.

### Claim 4 (depends on 1)

The method for detecting and classifying defects as recited in claim 1, further comprising:
training the autoencoder by an optimization objective function, wherein the optimization objective function is |X−X′|1+λ|X−X′|2, wherein X is the image to be detected, X′ is the reconstructed image, λ is a weight with a value range of 0.1-10, |X−X′|1 is an L1 norm of a reconstructed error image between the image and the reconstructed image, and |X−X′2 is L2 norm of the reconstruction error image.

### Claim 5 (depends on 1)

The method for detecting and classifying defects as recited in claim 1, further comprising:
calculating the structural similarity values between the image to be detected and the number of template images marking defect categories respectively according to SSIM(x,y)=[l(x,y)]α[c(x,y)]β[s(x,y)]γ, wherein,
l
⁡
(
x
,
y
)
=
2
⁢
μ
x
⁢
μ
y
+
C
1
μ
x
2
+
μ
y
2
+
C
1
,
c
⁡
(
x
,
y
)
=
2
⁢
σ
x
⁢
σ
y
+
C
2
σ
x
2
+
σ
y
2
+
C
2
,
s
⁡
(
x
,
y
)
=
2
⁢
σ
xy
+
C
3
σ
x
⁢
σ
y
+
C
3
,
x is the image to be detected, y is the template image, and SSIM(x,y) is the structural similarity values between the image to be detected and the template image, l(x,y) is used to compare a brightness of x with a brightness of y, μx, μy represent an image mean of x and an image mean of y, and c(x,y) is used to compare an image contrast of x with an image contrast of y, σx, σy represent an image standard deviation of x and an image standard deviation of y, σxy is an image covariance between x and y, C1, C2, C3 are constants.

### Claim 6 (depends on 1)

The method for detecting and classifying defects as recited in claim 1, wherein the structural similarity value of the image to be detected and the template image is positively correlated with the similarity between the image to be detected and the template image.

### Claim 7 (depends on 1)

The method for detecting and classifying defects as recited in claim 1, further comprising:
in response that the image to be detected complies with a defect criterion of filtering out small noise reconstruction error, determining that the image to be detected has defects;
in response that the image to be detected does not comply with the defect judgment criterion, determining that the image to be detected to be determined to have no defect.

### Claim 8

An electronic device comprising:
a processor; and
a non-transitory storage medium coupled to the processor and configured to store a plurality of instructions, which cause the processor to:
input an image to be detected to a trained autoencoder to obtain a reconstructed image corresponding to the image to be detected;
determine whether the image to be detected has defects based on a defect criteria for filtering out small noise reconstruction errors;
in response that the image to be detected has defects, calculate a plurality of structural similarity values between the image to be detected and a plurality of template images marking defect categories respectively;
determine a target defect category corresponding to the template image with the highest structural similarity value, and classifying the image to be detected into the target defect category, wherein a mathematical expression of the defect criteria is
∑
i
,
j
⁢
Δ
⁢
⁢
X
i
,
j
*
δ
⁢
⁢
X
i
,
j
∑
i
,
j
⁢
δ
⁢
⁢
X
i
,
j
>
τ
,
wherein ΔXi,j is a reconstruction error image between the image to be detected and the reconstructed image, δXi,j is a binary image of the reconstruction error image, i and j represent pixel positions, τ is a preset reconstruction error measurement threshold.

### Claim 9 (depends on 8)

The electronic device as recited in claim 8, wherein τ is a statistical value of a reconstruction error corresponding to a reconstructed error image between the image and the reconstructed image, and the t is adjusted based on a recall rate of a preset defect detection and an accuracy rate of the preset defect detection, the accuracy rate is a number of the images correctly detected as defective as a proportion of the number of all images detected as defective, and the recall rate is the number of the images correctly detected as defective as a proportion of the number of all images showing real defects.

### Claim 10 (depends on 8)

The electronic device as recited in claim 8, wherein a binary image calculation formula of the binary image is
δ
⁢
⁢
X
i
,
j
=
{
1
Δ
⁢
⁢
X
i
,
j
>
ɛ
0
otherwise
,
ε is a filtering threshold of small noise reconstruction error.

### Claim 11 (depends on 8)

The electronic device as recited in claim 8, wherein the plurality of instructions are further configured to cause the processor to:
train the autoencoder by an optimization objective function, wherein the optimization objective function is |X−X′|1+λ|X−X′|2, wherein X is the image to be detected, X′ is the reconstructed image, λ is a weight with a value range of 0.1-10, |X−X′|1 is an L1 norm of a reconstructed error image between the image and the reconstructed image, and |X−X′2 is L2 norm of the reconstruction error image.

### Claim 12 (depends on 8)

The electronic device as recited in claim 8, wherein the plurality of instructions are further configured to cause the processor to:
calculate the structural similarity values between the image to be detected and the number of template images marking defect categories respectively according to SSIM (x,y)=[l(x,y)]α[c(x,y)]β[s(x,y)]γ, wherein,
l
⁡
(
x
,
y
)
=
2
⁢
μ
x
⁢
μ
y
+
C
1
μ
x
2
+
μ
y
2
+
C
1
,
c
⁡
(
x
,
y
)
=
2
⁢
σ
x
⁢
σ
y
+
C
2
σ
x
2
+
σ
y
2
+
C
2
,
s
⁡
(
x
,
y
)
=
2
⁢
σ
xy
+
C
3
σ
x
⁢
σ
y
+
C
3
,
x is the image to be detected, y is the template image, and SSIM(x,y) is the structural similarity values between the image to be detected and the template image, l(x,y) is used to compare a brightness of x with a brightness of y, μx, μy represent an image mean of x and an image mean of y, and c(x,y) is used to compare an image contrast of x with an image contrast of y, σx, σy represent an image standard deviation of x and an image standard deviation of y, σxy is an image covariance between x and y, C1, C2, C3 are constants.

### Claim 13 (depends on 8)

The electronic device as recited in claim 8, wherein the structural similarity value of the image to be detected and the template image is positively correlated with the similarity between the image to be detected and the template image.

### Claim 14 (depends on 8)

The electronic device as recited in claim 8, wherein the plurality of instructions are further configured to cause the processor to:
in response that the image to be detected complies with a defect criterion of filtering out small noise reconstruction error, determine that the image to be detected has defects;
in response that the image to be detected doesn't comply with the defect judgment criterion, determine that the image to be detected to be determined to have no defect.

### Claim 15

A non-transitory storage medium having stored thereon instructions that, when executed by at least one processor of an electronic device, causes the least one processor to execute instructions of a method for detecting and classifying defects, the method comprising:
inputting an image to be detected to a trained autoencoder to obtain a reconstructed image corresponding to the image to be detected;
determining whether the image to be detected has defects based on a defect criteria for filtering out small noise reconstruction errors;
in response that the image to be detected has defects, calculating a plurality of structural similarity values between the image to be detected and a plurality of template images marking defect categories respectively;
determining a target defect category corresponding to the template image with the highest structural similarity value, and classifying the image to be detected into the target defect category, wherein a mathematical expression of the defect criteria is
∑
i
,
j
⁢
Δ
⁢
⁢
X
i
,
j
*
δ
⁢
⁢
X
i
,
j
∑
i
,
j
⁢
δ
⁢
⁢
X
i
,
j
>
τ
,
wherein ΔXi,j is a reconstruction error image between the image to be detected and the reconstructed image, δXi,j is a binary image of the reconstruction error image, i and j represent pixel positions, τ is a preset reconstruction error measurement threshold.

### Claim 16 (depends on 15)

The non-transitory storage medium as recited in claim 15, wherein τ is a statistical value of a reconstruction error corresponding to a reconstructed error image between the image and the reconstructed image, and the t is adjusted based on a recall rate of a preset defect detection and an accuracy rate of the preset defect detection, the accuracy rate is a number of the images correctly detected as defective as a proportion of the number of all images detected as defective, and the recall rate is the number of the images correctly detected as defective as a proportion of the number of all images showing real defects.

### Claim 17 (depends on 15)

The non-transitory storage medium as recited in claim 15, wherein a binary image calculation formula of the binary image is
δ
⁢
⁢
X
i
,
j
=
{
1
Δ
⁢
⁢
X
i
,
j
>
ɛ
0
otherwise
,
ε is a filtering threshold of small noise reconstruction error.