Method and apparatus for augmented data anomaly detection
AssigneeMorgan State University
InventorsMariem Ben Fadhel, Kofi Nyarko
Life of this patent
Estimating the term…
Abstract
A data anomaly detection method and apparatus in which a deep neural network is trained on baseline data. Sequences of statistics of each layer of the deep neural network are saved, processed and used to train an LSTM autoencoder across a variety of reconstruction error thresholds, and a preferred threshold is selected for an optimized autoencoder. In an Inference mode, a data sample is presented to the autoencoder; the reconstruction error is calculated and compared to the threshold. If it is above the threshold, then the data sample is an out-of-distribution sample, and the sample is tagged as anomalous.
Claims13
A method for detection of data anomalies via a deep multi-layer neural network architecture, the method being implemented by a computer system that comprises one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:
- in a neural network training phase:
- a. obtaining a first collection of actual data items corresponding to one or more groups of data categories, said first collection of actual data items having a first data distribution;
- b. using a first neural network to generate a set of synthetic data items using a synthetic data generation configuration;
- c. providing said collection of actual data items and said set of synthetic items to a second neural network;
- d. using the second neural network to (i) make a classification determination using a set of classification determination configurations including whether each data item in said set of synthetic data items is synthetic or actual, and (ii) update said set of classification determination configurations;
- e. providing said classification determinations to said first neural network;
- f. using said classification determinations by said first neural network to update said synthetic data generation configuration;
- g. repeating steps b through f until said second neural network cannot make a valid classification determination;
- h. generating autoencoder training sequences of updated classification determination configurations for each layer in said second neural network;
- in an autoencoder phase:
- i. providing said autoencoder training sequences to an autoencoder, and said autoencoder training itself to differentiate anomalous data from real data using said autoencoder training sequences across a range of reconstruction error thresholds;
- j. selecting a preferred reconstruction error threshold based on autoencoder performance during said training step to result in said autoencoder being optimized for recognition of anomalous data;
- in a data anomaly detection phase:
- k. submitting to the second neural network a purported data item;
- l. generating by said second neural network new sequences of classification determination configurations corresponding to said purported data item;
- m. providing said new sequences to said autoencoder, said autoencoder generating a prediction as to whether said purported data item falls within said first data distribution;
- n. classifying by said autoencoder said purported data item as anomalous if said purported data item falls outside said first data distribution;
- o. sending said new sequences to said second neural network if said purported data item is determined by said autoencoder to fall within said first data distribution, and making a classification determination by said second neural network for said purported data items using said set of classification configurations;
- and
- p. notifying a user that said purported data item may be anomalous if said second neural network determines that said purported data item is synthetic.
A method according to claim 1, wherein said first neural network and said second neural network are a generator and a discriminator, respectively, of a generative adversarial network.
A system comprising:
- a computer system that comprises one or more processors executing computer program instructions that, when executed, cause the computer system to:
- in a neural network training phase:
- a. obtain a first collection of actual data items corresponding to one or more groups of data categories, said first collection of actual data items having a first data distribution;
- b. use a first neural network to generate a set of synthetic data items using a synthetic data generation configuration;
- c. provide said collection of actual data items and said set of synthetic items to a second neural network;
- d. use the second neural network to (i) make a classification determination using a set of classification determination configurations including whether each data item in said set of synthetic data items are synthetic or actual, and (ii) update said set of classification determination configurations;
- e. provide said classification determinations to said first neural network;
- f. use said classification determinations by said first neural network to update said synthetic data generation configuration;
- g. repeat steps b through f until said second neural network cannot make a valid classification determination;
- h. generating autoencoder training sequences of updated classification determination configurations for each layer in said second neural network;
- in an autoencoder training phase:
- i. provide said autoencoder training sequences to an autoencoder to train itself to differentiate anomalous data from real data using said autoencoder training sequences across a range of reconstruction error thresholds;
- j. select a preferred reconstruction error threshold based on autoencoder performance during said training step to result in said autoencoder being optimized for recognition of anomalous data;
- in a data anomaly detection phase:
- k. submit to the second neural network a purported data item;
- l. generate by said second neural network new sequences of classification determination configurations corresponding to said purported data item;
- m. provide said new sequences to said autoencoder, and generate by said autoencoder a prediction as to whether said purported data item falls within said first data distribution;
- n. classify by said autoencoder said purported data item as anomalous if said purported data item falls outside said first data distribution;
- o. send said new sequences to said second neural network if said purported data item is determined by said autoencoder to fall within said first data distribution, and make a classification determination by said second neural network for said purported data item using said set of classification configurations;
- p. notify a user that said purported data item may be anomalous or malicious if said second neural network determines that said purported data item is synthetic.
A system according to claim 4, wherein said first neural network and said second neural network are a generator and a discriminator, respectively of a generative adversarial network.
An apparatus comprising:
- a first neural network configured to
- a. generate a set of synthetic data items using a synthetic data generation configuration;
- and
- b. provide a collection of actual text data items and said set of synthetic items to a second neural network, said collection of actual text data items having a first data distribution;
- a second neural network configured to
- (i) make a classification determination using a set of classification determination configurations whether each data item in said set of synthetic data items are synthetic or actual data,
- (ii) make a classification determination for each data item in said set of synthetic data items and said collection of actual data items using a set of classification configurations;
- and
- (iii) update said set of classification determination configurations;
- (iv) provide said classification determinations to said first neural network;
- said first neural network further configured to:
- c. use said classification determinations by said second neural network to update said synthetic data generation configuration;
- said second neural network further configured to:
- (v) generate autoencoder training sequences of updated classification determination configurations for each layer in said second neural network,
- said system further comprising an autoencoder configured to
- 1) use auto encoder training sequences to train itself to differentiate anomalous data from real data across a range of reconstruction error thresholds;
- 2) select a preferred reconstruction error threshold based on autoencoder performance to result in said autoencoder being optimized for recognition of anomalous data;
- said second neural network further configured to:
- (vi) generate new sequences of classification determination configurations corresponding to a purported data item and provide said new sequences to said autoencoder;
- said autoencoder further configured to:
- 3) generate a prediction using said new sequences as to whether said purported data item falls within said first data distribution;
- 4) classify said purported data item as anomalous if said purported data item falls outside said first data distribution;
- 5) send said new sequences to said second neural network if said purported data item is determined to fall within said first data distribution.
A method for detection of data anomalies via a deep multi-layer neural network architecture, the method being implemented by a computer system that comprises one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:
- a. training a semi-supervised neural network on a set of baseline data;
- b. saving and processing sequences of statistics generated during said training step for each layer of the neural network;
- c. training and validating an LSTM autoencoder using at least a portion of said processed sequences of statistics across a range of reconstruction error thresholds;
- d. examining a data sample by the LSTM autoencoder and calculating the reconstruction error by the autoencoder and comparing the reconstruction error of to a selected reconstruction error threshold;
- e. identifying said data sample as anomalous if the reconstruction error is above the selected reconstruction error threshold.
Dependencies are inferred from the claim text.