# US 2023/0068908 A1 · Anomaly Detection and Diagnosis in Factory Automation System using Pre-Processed Time-Delay Neural Network with Loss Function Adaptation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0068908 A1
- Document ID: US20230068908A1
- Status: Published application
- Kind: A1
- Application number: 17/464,901
- Filed: 2021-09-02
- Published: 2023-03-02
- Earliest claimed: 2021-09-02
- Assignees: Mitsubishi Electric Research Laboratories, Inc.
- Applicants: Mitsubishi Electric Research Laboratories, Inc.
- Inventors: Jianlin Guo; Bryan Liu; Toshiaki Koike Akino; Ye Wang; Kyeong Jin Kim; Kieran Parsons; Philip Orlik
- Web page: https://patentgrep.com/patent/US20230068908A1

## CPC

- G05B 23/0283: {Predictive maintenance, e.g. involving the monitoring of a system and, based on the monitoring results, taking decisions on the maintenance schedule of the monitored system; Estimating remaining useful life [RUL] (preventive maintenance, i.e. planning maintenance according to the available resources without monitoring the system G06Q10/06)}
- G05B 19/4184: {characterised by fault tolerance, reliability of production system}
- G05B 23/024: {Quantitative history assessment, e.g. mathematical relationships between available data; Functions therefor; Principal component analysis [PCA]; Partial least square [PLS]; Statistical classifiers, e.g. Bayesian networks, linear regression or correlation analysis; Neural networks}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G05B 23/0221: {Preprocessing measurements, e.g. data collection rate adjustment; Standardization of measurements; Time series or signal analysis, e.g. frequency analysis or wavelets; Trustworthiness of measurements; Indexes therefor; Measurements using easily measured parameters to estimate parameters difficult to measure; Virtual sensor creation; De-noising; Sensor fusion; Unconventional preprocessing inherently present in specific fault detection methods like PCA-based methods}
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented pre-processed time-delay autoencoder based anomaly detection method are provided for detecting anomalous states of machines arranged in a factory automation (FA) system or a manufacturing production line. The method includes acquiring source signals from the machines via an interface performing a data pre-processing process for the acquired source signals by normalizing value ranges of the acquired source signals and filtering undesired features from the acquired source signals performing a time-delayed data reform process for the pre-processed source signals based on a time-delay window to generate pre-processed time-delay data submitting pre-processed time-delay testing data to a pre-processed time-delayed autoencoder (Prep-TDAE) neural network, wherein the pre-processed time-delay testing data are collected online while the machines are operated, wherein the Prep-TDAE neural network has been pre-trained by using the pre-processed time-delay training data detecting, if an anomaly state is encountered with respect to the machines, by computing anomaly scores of the pre-processed time-delay testing data, and determining, when the anomaly state is detected, anomaly occurrence time, duration and severity with respect to the anomaly state of each of the machines.

## Claims

### Claim 1

A computer-implemented pre-processed time-delay autoencoder based anomaly detection method for detecting anomalous states of machines arranged in a factory automation (FA) system or a manufacturing production line, comprising steps of:
acquiring source signals from the machines via an interface;
performing a data pre-processing process for the acquired source signals by normalizing value ranges of the acquired source signals and filtering undesired features from the acquired source signals;
performing a time-delayed data reform process for the pre-processed source signals based on a time-delay window to generate pre-processed time-delay data;
submitting pre-processed time-delay testing data to a pre-processed time-delayed autoencoder (Prep-TDAE) neural network, wherein the pre-processed time-delay testing data are collected online while the machines are operated, wherein the Prep-TDAE neural network has been pre-trained by using the pre-processed time-delay training data;
detecting, if an anomaly state is encountered with respect to the machines, by computing anomaly scores of the pre-processed time-delay testing data; and
determining, when the anomaly state is detected, anomaly occurrence time, duration and severity with respect to the anomaly state of each of the machines.

### Claim 2 (depends on 1)

The method of claim 1, wherein the data pre-processing process divides the normalized source signals into normalized analog signals and normalized binary signals.

### Claim 3 (depends on 1)

The method of claim 1, wherein in the determining step, the anomaly state is detected by computing the anomaly score of the normalized signals.

### Claim 4 (depends on 1)

The method of claim 1, wherein an input later of the Prep-TDAE neural network is configured to accept binary part of the pre-processed time-delay data via a binary acceptor of the input layer and accept analog part of the pre-processed time-delay data via an analog acceptor of the input layer.

### Claim 5 (depends on 1)

The method of claim 1, wherein in the step of pre-processing, the acquired signals are divided into binary signals and analog signals, wherein the analog signals are processed by a band-pass filter followed by a high-pass filter, wherein the binary signals are processed by a unit impulse filter.

### Claim 6 (depends on 1)

The method of claim 1, wherein the Prep-TDAE neural network performs loss function adaptation by applying different loss functions to different data types.

### Claim 7 (depends on 6)

The method of claim 6, wherein the binary signals are trained with cross-entropy loss function and the analog signals are trained with squared error loss function.

### Claim 8 (depends on 1)

The method of claim 1, wherein the anomaly occurrence time corresponds to time index of test data, at which anomaly score exceeds a predefined threshold.

### Claim 9 (depends on 1)

The method of claim 1, wherein the anomaly duration is the length of the time during which the anomaly score of test data persistently exceeds a predefined threshold for the consecutive time indices.

### Claim 10 (depends on 1)

The method of claim 1, wherein the anomaly severity is determined based on a magnitude of the anomaly score.

### Claim 11

A computer-implemented statistical-based anomaly diagnosis method for diagnosing anomaly sources of machines arranged in a factory automation (FA) system or a manufacturing production line, comprising steps of:
acquiring source signals from the machines via an interface;
performing a data pre-processing process for the acquired source signals based on a data pre-processing process algorithm;
performing a time-delayed data reform process algorithm for the acquired source signals based on a time-delay window to generate pre-processed time-delay data;
normalizing the pre-processed time-delay data so as to have zero mean and unit variance;
computing a statistical shrunk covariance matrix to avoid the covariance matrix to be noninvertible;
computing a whitening matrix for whitening transform;
performing the whitening transform of the signals for the mean and the variance computation;
computing the mean and the variance of the transformed signal for an anomaly score computation;
computing an anomaly score for each data signal for anomalous data signal diagnosis;
diagnosing the anomalous data signals; and
identifying a machine with the anomaly among the machines using the anomaly diagnosis algorithm.

### Claim 12 (depends on 11)

The method of claim 11, wherein the anomaly score for the given normalized testing sequence {acute over (x)}Tt=[{acute over (x)}Pt, . . . , {acute over (x)}Pt+w-1] is computed in the following steps:
computing the anomaly score in a pre-processed form pt=[pFt,pBt,pHt]=Σm=tt+w-1({acute over (x)}Pm)2, wherein ({acute over (x)}Pm)2 is the pre-processed form of the anomaly score at the mth time index;
computing the anomaly score aAt of the analog sensors are computed as aAt=[pBt+pHt]; and
computing the final anomaly score of sensors is computed as at=[pFt,aAt].

### Claim 13

A computer-implemented residual-based anomaly diagnosis method for diagnosing anomaly sources of machines arranged in a factory automation (FA) system or a manufacturing production line, comprising steps of:
acquiring source signals from the machines via an interface;
performing a data pre-processing process algorithm for the acquired source signals
performing a time-delayed data reform process algorithm for the acquired source signals based on a time-delay window to generate pre-processed time-delay data;
submitting the pre-processed time-delay data to a pre-processed time-delay autoencoder (Pre-TDAE) neural network to generate reconstructed signals, wherein the Prep-TDAE neural network has been pre-trained by training data;
computing a residual error based on the reconstructed signals;
computing mean and variance of the residual error;
computing anomaly scores for the reconstructed signals;
diagnosing anomalous signals corresponding to the source signals of the machines based on the anomaly scores; and
identifying a machine with an anomaly among the machines using an anomaly diagnosis algorithm.

### Claim 14 (depends on 13)

The method of claim 13, wherein the anomaly score for the given normalized residual errors ŕTt=[ŕPt, . . . , ŕPt+w-1] is computed following the steps below:
Computing the anomaly score in a pre-processed form pt=[pFt,pBt,pHt]=Σm=tt+w-1 abs(ŕPm), where abs(rPm) is the pre-processed form of the anomaly score at the mth time index:
Computing the anomaly score aAt of the analog sensors are computed as aAt=[pBt+pHt]:
Computing the final anomaly score of sensors is as at=[pFt,aAt].

### Claim 15 (depends on 13)

The method of claim 13, wherein the residual error the for the binary signal xT,it is computed as
r
T
,
i
t
=
-
x
T
,
i
t
⁢
log
⁢
(
1
1
+
exp
⁡
(
-
x
ˆ
T
,
i
t
)
)
⁢
(
1
-
X
T
,
i
t
)
⁢
log
⁢
(
1
1
+
exp
⁡
(
-
x
ˆ
T
,
i
t
)
)
.

### Claim 16 (depends on 13)

The method of claim 13, wherein the residual error for the analog signal xT,jt is computed as
rT,jt=½(xT,jt−{circumflex over (x)}T,jt)2.

### Claim 17 (depends on 1)

A pre-processed time-delay autoencoder based anomaly detection system for detecting anomalous states of machines arranged in a factory automation (FA) system or a manufacturing production line, comprising steps of:
an interface configured to acquire source signals from the machines;
a memory to store computer-executable programs including a data pre-processing algorithm, a time-delayed data reform process algorithm, a pre-processed time-delay autoencoder (Pre-TDAE) neural network, an anomaly diagnosis algorithm and a computer-implemented statistical-based anomaly diagnosis method of claim 1; and
a processor configured to perform steps of:
performing the data pre-processing process algorithm for the acquired source signals by normalizing value ranges of the acquired source signals and filtering undesired features from the acquired source signals;
performing the time-delayed data reform process algorithm for the pre-processed source signals based on a time-delay window to generate pre-processed time-delay data;
submitting pre-processed time-delay testing data to the Prep-TDAE neural network, wherein the pre-processed time-delay testing data are collected online while the machines are operated, wherein the Prep-TDAE neural network has been pre-trained by using the pre-processed time-delay training data;
detecting, if an anomaly state is encountered with respect to the machines, by computing anomaly scores of the pre-processed time-delay testing data; and
determining, when the anomaly state is detected, anomaly occurrence time, duration and severity with respect to the anomaly state of each of the machines.

### Claim 18 (depends on 17)

The system of claim 17, wherein the data pre-processing process divides the normalized source signals into normalized analog signals and normalized binary signals.

### Claim 19 (depends on 11)

A statistical-based anomaly diagnosis system for diagnosing anomaly sources of machines arranged in a factory automation (FA) system or a manufacturing production line, comprising:
an interface configured to acquire source signals from the machines;
a memory to store computer-executable programs including a data pre-processing algorithm, a time-delayed data reform process algorithm, an anomaly diagnosis algorithm and a computer-implemented statistical-based anomaly diagnosis method of claim 11; and
a processor configured to execute the computer-executable programs, wherein the processor performs steps of:
performing the data pre-processing process algorithm for the acquired source signals;
performing the time-delayed data reform process algorithm for the acquired source signals based on a time-delay window to generate pre-processed time-delay data;
normalizing the pre-processed time-delay data so as to have zero mean and unit variance;
computing a statistical shrunk covariance matrix to avoid the covariance matrix to be noninvertible;
computing s whitening matrix for whitening transform;
performing the whitening transform of the signals for the mean and the variance computation;
computing the mean and the variance of the transformed signal for a anomaly score computation;
computing an anomaly score for each data signal for anomalous data signal diagnosis;
diagnosing the anomalous data signals; and
identifying a machine with the anomaly among the machines using the anomaly diagnosis algorithm.

### Claim 20 (depends on 19)

The system of claim 19, wherein the anomaly score for the given normalized testing sequence {acute over (x)}Tt=[{acute over (x)}Pt, . . . , {acute over (x)}Pt+w-1] is computed in the following steps:
computing the anomaly score in a pre-processed form pt=[pFt,pBt,pHt]=Σm=tt+w-1({acute over (x)}Pm)2, wherein ({acute over (x)}Pm)2 is the pre-processed form of the anomaly score at the mth time index;
computing the anomaly score aAt of the analog sensors are computed as aAt=[pBt+pHt]; and
computing the final anomaly score of sensors is computed as at=[pFt,aAt].

### Claim 21 (depends on 13)

A residual-based anomaly diagnosis system for diagnosing anomaly sources of machines arranged in a factory automation (FA) system or a manufacturing production line, comprising:
an interface configured to acquire source signals from the machines;
a memory to store computer-executable programs including a data pre-processing process algorithm, a time-delayed data reform process algorithm, a pre-processed time-delay autoencoder (Pre-TDAE) neural network, an anomaly diagnosis algorithm and a computer-implemented residual-based anomaly diagnosis method of claim 13;
a processor configured to execute the computer-executable programs, wherein the processor is configured to perform steps of:
performing the data pre-processing process algorithm for the acquired source signals;
performing the time-delayed data reform process algorithm for the acquired source signals based on a time-delay window to generate pre-processed time-delay data;
submitting the pre-processed time-delay data to the Pre-TDAE neural network to generate reconstructed signals, wherein the Prep-TDAE neural network has been pre-trained by training data;
computing a residual error based on the reconstructed signals;
computing mean and variance of the residual error;
computing anomaly scores for each data signal;
diagnosing the anomalous signals corresponding to the source signals of the machines based on the anomaly scores; and
identifying a machine with an anomaly among the machines using the anomaly diagnosis algorithm.

### Claim 22 (depends on 21)

The system of claim 21, wherein the anomaly score for the given normalized residual errors ŕTt=[ŕPt, . . . , ŕPt+w-1] is computed following the steps below:
Computing the anomaly score in a pre-processed form pt=[pFt,pBt,pHt]=Σm=tt+w-1 abs(ŕPm), where abs(rPm) is the pre-processed form of the anomaly score at the mth time index.
Computing the anomaly score aAt of the analog sensors are computed as aAt=[pBt+pHt].
Computing the final anomaly score of sensors is as at=[pFt,aAt].

### Claim 23 (depends on 21)

The system of claim 21, wherein the residual error the for the binary signal xT,it is computed as
r
T
,
i
t
=
-
x
T
,
i
t
⁢
log
⁡
(
1
1
+
exp
⁡
(
-
x
ˆ
T
,
i
t
)
)
⁢
(
1
-
X
T
,
i
t
)
⁢
log
⁢
(
1
1
+
exp
⁡
(
-
x
ˆ
T
,
i
t
)
)
.

### Claim 24 (depends on 21)

The system of claim 21, wherein the residual error for the analog signal xT,jt is computed as
rT,jt=½(xT,jt−{circumflex over (x)}T,jt)2.
