# US 2022/0277017 A1 · Standardization in the context of data integration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277017 A1
- Document ID: US20220277017A1
- Status: Published application
- Kind: A1
- Application number: 17/183,915
- Filed: 2021-02-24
- Published: 2022-09-01
- Earliest claimed: 2021-02-24
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Namit Kabra; Krishna Kishore Bonagiri; Mike W. Grasselt; Yannick Saillet
- Web page: https://patentgrep.com/patent/US20220277017A1

## CPC

- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/213: {with details for schema evolution support}
- G06F 16/256: {in federated or virtual databases}
- G06F 16/951: Indexing; Web crawling techniques
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 30/1983: {Syntactic or structural pattern recognition, e.g. symbolic string recognition}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques are described relating to automatic data standardization in a managed services domain of a cloud computing environment. An associated computer-implemented method includes receiving a dataset during a data onboarding procedure and classifying datapoints within the dataset. The method further includes applying a machine learning data standardization model to each classified datapoint within the dataset and deriving a proposed set of data standardization rules for the dataset based upon any standardization modification determined consequent to model application. Optionally, the method includes presenting the proposed set of data standardization rules for client review and, responsive to acceptance of the proposed set of data standardization rules, applying the proposed set of data standardization rules to the dataset. The method further includes, responsive to acceptance of the proposed set of data standardization rules, updating the machine learning data standardization model accordingly.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving a dataset during a data onboarding procedure;
classifying datapoints within the dataset;
applying a machine learning data standardization model to each classified datapoint within the dataset; and
deriving a proposed set of data standardization rules for the dataset based upon any standardization modification determined consequent to application of the machine learning data standardization model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
presenting the proposed set of data standardization rules for client review; and
responsive to acceptance of the proposed set of data standardization rules, applying the proposed set of data standardization rules to the dataset.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
responsive to acceptance of the proposed set of data standardization rules, updating the machine learning data standardization model based upon the proposed set of data standardization rules.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein configuring the machine learning data standardization model comprises:
sampling a plurality of datasets during a plurality of respective data onboarding scenarios;
identifying respective data anomaly scenarios based upon the plurality of sampled datasets;
identifying existing data anomaly rectification techniques to address the respective data anomaly scenarios; and
training the machine learning data standardization model based upon applicability of the existing data anomaly rectification techniques to the respective data anomaly scenarios.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein applying the machine learning data standardization model to each classified datapoint within the dataset comprises:
responsive to determining that the datapoint is an outlier value addressed by a preexisting standardization rule incorporated into the machine learning data standardization model, dynamically modifying the outlier value by applying the preexisting standardization rule.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein applying the machine learning data standardization model to each classified datapoint within the dataset comprises:
responsive to determining that the datapoint is an outlier value unaddressed by any preexisting standardization rule, determining at least one standardization modification to remediate the datapoint.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein determining at least one standardization modification to remediate the datapoint comprises:
responsive to determining that the datapoint is a null value, determining a replacement for the null value based upon a relationship between a first data column including the null value and at least one correlated data column or based upon a relationship between interrelated values within the first data column.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6, wherein determining at least one standardization modification to remediate the datapoint comprises:
responsive to determining that the datapoint is a frequent outlier string value having an occurrence rate of greater than or equal to a predetermined dataset frequency threshold, categorizing the frequent outlier string value as a valid value.

### Claim 9 (depends on 6)

The computer-implemented method of claim 6, wherein determining at least one standardization modification to remediate the datapoint comprises:
responsive to determining that the datapoint is an infrequent outlier string value having an occurrence rate of less than a predetermined dataset frequency threshold, applying a data crawling algorithm to evaluate the infrequent outlier string value via an automated application, wherein inputs to the data crawling algorithm include the infrequent outlier string value and an associated data class name.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein applying the data crawling algorithm to evaluate the infrequent outlier string value comprises:
responsive to identifying a threshold quantity of datapoints associated with both the infrequent outlier string value and the associated data class name, categorizing the infrequent outlier string value as a valid value.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein applying the data crawling algorithm to evaluate the infrequent outlier string value comprises:
responsive to failure to identify a threshold quantity of datapoints associated with both the infrequent outlier string value and the associated data class name, determining whether there is at least one valid string value within the dataset having a predefined degree of string similarity to the infrequent outlier string value; and
responsive to identifying at least one valid string value within the dataset having the predefined degree of string similarity, determining a remediation for the infrequent outlier string value by selecting a valid string value among the at least one valid string value and determining at least one modification to the infrequent outlier string value based upon any identified character difference between the infrequent outlier string value and the selected valid string value.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein selecting a valid string value among the at least one valid string value comprises assessing quantitative similarity between the infrequent outlier string value and each of the at least one valid string value.

### Claim 13 (depends on 11)

The computer-implemented method of claim 11, wherein selecting a valid string value among the at least one valid string value comprises applying one or more heuristics based at least in part upon string value selection history.

### Claim 14 (depends on 11)

The computer-implemented method of claim 11, wherein applying the data crawling algorithm to evaluate the infrequent outlier string value further comprises:
responsive to failure to identify at least one valid string value within the dataset having the predefined degree of string similarity, applying at least one crowdsourcing technique to determine a remediation for the infrequent outlier string value.

### Claim 15 (depends on 6)

The computer-implemented method of claim 6, wherein determining at least one standardization modification to remediate the datapoint comprises:
responsive to determining that the datapoint is an outlier non-string value, determining a remediation for the outlier non-string value.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein determining a remediation for the outlier non-string value comprises:
identifying a regular expression format related to the outlier non-string value; and
determining at least one modification to the outlier non-string value to conform with the regular expression format.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15, wherein determining a remediation for the outlier non-string value comprises:
parsing the outlier non-string value into at least one string portion; and
for each outlier string portion among the at least one string portion:
selecting a valid string value among at least one valid string value by assessing quantitative similarity between the outlier string portion and each of the at least one valid string value; and
determining at least one modification to the outlier string portion based upon any identified character difference between the outlier string portion and the selected valid string value.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, wherein determining a remediation for the outlier non-string value comprises:
responsive to determining that the outlier non-string value is a duplicate value, determining a replacement for the duplicate value based upon a relationship between a first data column including the duplicate value and at least one correlated data column.

### Claim 19

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:
receive a dataset during a data onboarding procedure;
classify datapoints within the dataset;
apply a machine learning data standardization model to each classified datapoint within the dataset; and
derive a proposed set of data standardization rules for the dataset based upon any standardization modification determined consequent to application of the machine learning data standardization model.

### Claim 20

A system comprising:
at least one processor; and
a memory storing an application program, which, when executed on the at least one processor, performs an operation comprising:
receiving a dataset during a data onboarding procedure;
classifying datapoints within the dataset;
applying a machine learning data standardization model to each classified datapoint within the dataset; and
deriving a proposed set of data standardization rules for the dataset based upon any standardization modification determined consequent to application of the machine learning data standardization model.