# US 2020/0349169 A1 · Artificial intelligence (ai) based automatic data remediation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0349169 A1
- Document ID: US20200349169A1
- Status: Published application
- Kind: A1
- Application number: 16/403,153
- Filed: 2019-05-03
- Published: 2020-11-05
- Earliest claimed: 2019-05-03
- Assignees: Accenture Global Solutions Limited
- Applicants: Accenture Global Solutions Limited
- Inventors: Mahesh VENKATESAN; Akshay BHAIYA; Mrinal SHANKAR
- Web page: https://patentgrep.com/patent/US20200349169A1

## CPC

- G06N 20/20: Ensemble learning
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An Artificial Intelligence (AI)-based data remediation system accesses analyzes a dataset for identification and remediation of anomalies including missing values and outliers. Based on the attributes associated with the anomalies, the dataset is initially processed via statistical checks for identifying if the dataset includes one of point anomalies, contextual anomalies and time trend anomalies. Based on the type of anomalies various machine learning (ML) based techniques can be employed for identification of the anomalous data points. Expected values are generated. The anomalous data points are replaced with the expected values for the generation of a transformed dataset.

## Claims

### Claim 1

An Artificial Intelligence (AI) based automatic data remediation system comprising:
at least one processor;
a non-transitory processor readable medium storing machine-readable instructions that cause the at least one processor to:
access a dataset stored to a data lake from one or more of a plurality of data sources;
receive a user input conveying a number of attributes of the dataset;
determine whether the dataset is to be treated for one or more anomalies including missing values and outliers in the dataset via executing a statistical check on the dataset;
identify portions of the dataset that include the anomalies based on the statistical check;
classify the anomalies into one of a plurality of anomaly types that include point anomalies, contextual anomalies and time trend anomalies, the anomaly types being determined based on the number of attributes of the dataset;
identify anomalous data points in the dataset that correspond to the anomalies, wherein the determination of the anomalous data points is based on the anomaly type detected in the dataset;
obtain expected values for the anomalous data points, the expected values being obtained by employing autoencoders;
generate a transformed dataset from the dataset by replacing the data points corresponding to the anomalies in the dataset with the expected values; and
upload the transformed dataset to the data lake.

### Claim 2 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine whether the anomalies include one of a plurality of anomaly types further comprises instructions that cause the processor to:
determine that the dataset includes the point anomalies when the user input indicates that the dataset has only one attribute.

### Claim 3 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine whether the anomalies include one of a plurality of anomaly types further comprises instructions that cause the processor to:
determine that the dataset includes the contextual anomalies when the user input indicates that the dataset has multiple attributes.

### Claim 4 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine whether the anomalies include one of a plurality of anomaly types further comprises instructions that cause the processor to:
determine that the dataset includes the time trend anomalies when the user input indicates that the dataset tracks a single attribute over a time period.

### Claim 5 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine whether the dataset is to be treated for one or more anomalies via the statistical check further comprises instructions that cause the processor to:
compare a distribution of the dataset with a standard distribution of the dataset; and
determine that a percentage number of the anomalies in the dataset exceed a predetermined threshold.

### Claim 6 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine data points in the dataset that correspond to the anomalies further comprises instructions that cause the processor to:
when the anomalies include the point anomalies,
obtain a univariate distribution of the dataset;
determine a threshold from the univariate distribution for identifying the outliers; and
determine percentile distribution volatility and the outliers beyond the threshold.

### Claim 7 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine data points in the dataset that correspond to the anomalies further comprises instructions that cause the processor to:
when the anomalies include the contextual anomalies,
obtain a decision boundary by processing the dataset via isolation forest technique, and
classify the data points as the anomalies via a comparison with the decision boundary.

### Claim 8 (depends on 7)

The automatic data remediation system of claim 7, further comprises instructions that cause the processor to:
train the isolation forest model to obtain the decision boundary using training data for a domain pertaining to the dataset.

### Claim 9 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to determine data points in the dataset that correspond to the anomalies further comprises instructions that cause the processor to:
when the anomalies include time trend anomalies:
cluster the data points within the dataset using self-organized maps;
compare the clusters obtained from the dataset to clusters obtained from a dataset of a previous time period; and
identify one or more of the data points changing clusters from the previous time period as the anomalies.

### Claim 10 (depends on 9)

The automatic data remediation system of claim 9, further comprises instructions that cause the processor to:
train the self-organized maps to cluster the data points of the dataset using training data for a domain pertaining to the dataset.

### Claim 11 (depends on 1)

The automatic data remediation system of claim 1 wherein the instructions to obtain the expected values for the data points that correspond to the anomalies further comprises instructions that cause the processor to:
input the data points corresponding to the anomalies to an autoencoder; and
obtain the expected values corresponding to each of the input data points from the autoencoder.

### Claim 12 (depends on 1)

The automatic data remediation system of claim 1 further comprising instructions that cause the processor to:
generate interactive visualizations displaying the anomalies.

### Claim 13 (depends on 1)

The automatic data remediation of claim 1 wherein the dataset pertains to finance domain and the non-transitory processor readable medium storing further machine-readable instructions that cause the at least one processor to:
flag the anomalies for fraudulent activity checks.

### Claim 14 (depends on 1)

The automatic data remediation of claim 1, wherein the dataset pertains to manufacturing domain and the non-transitory processor readable medium storing further machine-readable instructions that cause the at least one processor to:
flag the anomalies as system errors.

### Claim 15

An artificial intelligence (AI) based data remediation method comprising:
accessing a dataset stored to a data lake from one or more of a plurality of data sources;
receiving a user input conveying a number of attributes of the dataset;
determining whether the dataset is to be treated for one or more anomalies including missing values and outliers in the dataset via executing a statistical check on the dataset;
identifying portions of the dataset that include the anomalies based on the statistical check;
classifying the anomalies into one of a plurality of anomaly types that include point anomalies, contextual anomalies and time trend anomalies, the anomaly types being determined based on the number of attributes of the dataset;
identifying anomalous data points in the dataset that correspond to the anomalies, wherein the determination of the anomalous data points is based on the anomaly type detected in the dataset; and
outputting the anomalous data points via a user interface for user validation.

### Claim 16 (depends on 15)

The method of claim 15 wherein the dataset pertains to credit card balance and the user validation includes fraudulent activity check.

### Claim 17 (depends on 15)

The method of claim 15, wherein the user validation includes identity verification and the method further comprises:
obtaining expected values including corrected user identity for the anomalous data points, pertaining to incorrect identities, wherein the expected values are obtained by employing autoencoders;
generating a transformed dataset from the dataset by replacing the incorrect identities in the dataset with the corrected user identity; and
upload the transformed dataset to the data lake.

### Claim 18

A non-transitory processor-readable storage medium comprising machine-readable instructions that cause a processor to:
access a dataset stored to a data lake from one or more of a plurality of data sources;
receive a user input conveying a number of attributes of the dataset;
determine whether the dataset is to be treated for one or more anomalies including missing values and outliers in the dataset via executing a statistical check on the dataset;
identify portions of the dataset that include the anomalies based on the statistical check;
classify the anomalies into one of a plurality of anomaly types that include point anomalies, contextual anomalies and time trend anomalies, the anomaly types being determined based on the number of attributes of the dataset;
identify anomalous data points in the dataset that correspond to the anomalies, wherein the determination of the anomalous data points is based on the anomaly type detected in the dataset;
obtain expected values for the anomalous data points, the expected values being obtained by employing autoencoders;
generate a transformed dataset from the dataset by replacing the data points corresponding to the anomalies in the dataset with the expected values; and
upload the transformed dataset to the data lake.

### Claim 19 (depends on 18)

The non-transitory processor-readable storage medium of claim 18, further comprising instructions that cause the processor to:
input the data points corresponding to the anomalies to an autoencoder; and
obtain the expected values corresponding to each of the input data points from the autoencoder.

### Claim 20 (depends on 18)

The non-transitory processor-readable storage medium of claim 18, further comprising instructions that cause the processor to:
generate interactive visualizations displaying the anomalies.
