# US 2004/0083199 A1 · Method and architecture for data transformation, normalization, profiling, cleansing and validation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2004/0083199 A1
- Document ID: US20040083199A1
- Status: Published application
- Kind: A1
- Application number: 10/635,891
- Filed: 2003-08-05
- Published: 2004-04-29
- Earliest claimed: 2002-08-07
- Inventors: Diwakar R. Govindugari; David O. McGoveran
- Web page: https://patentgrep.com/patent/US20040083199A1

## CPC

- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

This is a computer software architecture and method for managing data transformation, normalization, profiling, cleansing, and validation. In the preferred embodiment, the architecture and method includes seven integrated functional elements: Dispatcher to route data and metadata among system elements; Semantic Modeler to build semantic models; Model Mapper to associate related concepts between semantic models; Transformation Manager to capture transformation rules and apply them to data driven by maps between semantic models; Validation Manager to capture data constraints and apply them to data; Interactive Guides to assist the processes of semantic modeling and semantic model mapping; and Adapters to convert data to and from specialized formats and protocols.

## Claims

### Claim 1

A computer implemented method for integrating data, said method comprising:
creating at least a first and a second semantic model wherein said first semantic model is restricted to a first category of knowledge and said second semantic model is restricted to a second category of knowledge;
storing said semantic models;
mapping the stored first semantic model to the stored second semantic model, thereby creating a model mapping;
storing said model mapping;
accepting as input a first data associated with said first semantic model;
transforming said first data, according to said model mapping;
validating said first data according to a set of validation rules; and,
forwarding said transformed and validated first data to at least a first software system.

### Claim 2 (depends on 1)

A method as in claim 1, wherein said step of mapping is further augmented with at least a third semantic model and said third semantic model is restricted to a third category of knowledge.

### Claim 3 (depends on 1)

A method as in claim 1, wherein said first and second categories of knowledge pertain to a common application domain.

### Claim 4 (depends on 3)

A method as in claim 3, wherein the common application domain is further modeled by at least one topic semantic model.

### Claim 5 (depends on 4)

A method as in claim 4, wherein at least a first topic is associated with the common application domain and the said association is maintained in a template.

### Claim 6 (depends on 5)

A method as in claim 5, wherein the template incorporates a second topic, relationships among the first and second topics, and at least one pre-defined rule.

### Claim 7 (depends on 2)

A method as in claim 2, wherein said third semantic model is a referent semantic model.

### Claim 8 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models describes the semantics of a message.

### Claim 9 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models describes the semantics of a Web Service.

### Claim 10 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models describes the semantics of a business document.

### Claim 11 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models describes the semantics of an XML document.

### Claim 12 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models describes the semantics of a database.

### Claim 13 (depends on 1)

A method as in claim 1, wherein the step of creating the semantic models may be augmented at the discretion of a human user by importing a set of semantic information.

### Claim 14 (depends on 13)

A method as in claim 13, wherein the set of semantic information is imported by means of a first adapter.

### Claim 15 (depends on 1)

A method as in claim 1, wherein the step of creating the semantic models includes user modification of at least one of the said semantic models.

### Claim 16 (depends on 1)

A method as in claim 1, wherein the step of creating the semantic models includes augmenting the semantic models indirectly with at least one validation rule.

### Claim 17 (depends on 1)

A method as in claim 1, wherein the step of creating the semantic models includes augmenting the semantic models indirectly with at least one transformation rule.

### Claim 18 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models is implemented as an ontology.

### Claim 19 (depends on 1)

A method as in claim 1, wherein at least one of the semantic models is represented by a standard knowledge description and querying language.

### Claim 20 (depends on 13)

A method as in claim 13, wherein the semantic information is processed according to at least a first rule in order to accomplish at least one of the operations of data profiling, semantic mapping, semantic resolution, data cleansing, normalization, transformation, and validation.

### Claim 21 (depends on 1)

A method as in claim 1, wherein said step of mapping the stored first semantic model to the stored second semantic model further comprises:
selecting and accessing said first semantic model based on association with a source;
selecting and accessing said second semantic model based on association with a destination;
presenting the semantic models to a user;
eliciting selection of a first semantic element belonging to the first semantic model;
eliciting selection of a second semantic element belonging to the second semantic model;
establishing an association between the first semantic element and the second semantic element;
providing the option of using system help as needed;
defining each relevant transformation rule;
defining each relevant validation rule;
providing the option of storing the resulting model mapping;
permitting editing of the association; and,
storing the model mapping.

### Claim 22 (depends on 21)

A method as in claim 21, where in the step of providing the option of using system help is accomplished using an Interactive Guide.

### Claim 23 (depends on 22)

A method as in claim 22, wherein the method implemented by said Interactive Guide comprises the steps of:
creating at least one candidate mapping between elements of said first semantic model and said second semantic model;
assigning a weight to each said candidate mapping, said weight derived from one or more portions that may be individually computed;
evaluating each candidate mapping and eliminating any candidate mapping that is invalid;
presenting a set of one or more candidate mappings to a human user;
eliciting from the user selection of at least one weighted candidate mapping in the set; and,
modifying the model mapping according to the user selection.

### Claim 24 (depends on 23)

A method as in claim 23, wherein the weight assigned to the candidate mapping is determined according to one or more heuristic rules, each of which determines a portion of said weight.

### Claim 25 (depends on 24)

A method as in claim 24, wherein at least one heuristic rule is defined the user.

### Claim 26 (depends on 24)

A method as in claim 24, wherein at least one heuristic rule is modified by a human user.

### Claim 27 (depends on 24)

A method as in claim 24, wherein a first heuristic rule is pre-defined and a criterion of applicability of the heuristic rule is determined by a human user.

### Claim 28 (depends on 23)

A method as in claim 23 wherein the system identifies those portions of the weight that cannot change on recalculation and does not recalculate them once they have been calculated.

### Claim 29 (depends on 23)

A method as in claim 23, wherein the inclusion of each candidate mapping in the set is decided based on the weight of that candidate mapping.

### Claim 30 (depends on 29)

A method as in claim 29, wherein the inclusion of each candidate mapping in the set is decided based on the weight of that candidate mapping exceeding a threshold.

### Claim 31 (depends on 30)

A method as in claim 30, wherein the threshold may be modified by the user.

### Claim 32 (depends on 23)

A method as in claim 23, wherein the number of candidate mappings included in the set is limited to a maximum number.

### Claim 33 (depends on 32)

A method as in claim 32, wherein the maximum number may be modified by the user.

### Claim 34 (depends on 23)

A method as in claim 23, wherein the user obtains an explanation of the weight of a selected candidate mapping was computed.

### Claim 35 (depends on 23)

A method as in claim 23, wherein the user may modify any portion of the weight.

### Claim 36 (depends on 23)

A method as in claim 23, wherein the user may modify the method by which the weight is derived.

### Claim 37 (depends on 1)

A method as in claim 1, wherein the means of accepting data is via an Adapter.

### Claim 38 (depends on 37)

A method as in claim 37, wherein the Adapter is a SOAP Message Handler.

### Claim 39 (depends on 1)

A method as in claim 1, wherein the means of forwarding data is via an Adapter.

### Claim 40 (depends on 39)

A method as in claim 39, wherein the Adapter is a SOAP Message Handler.

### Claim 41

A general-purpose computer incorporating specific hardware and software for transforming, profiling, cleansing, normalizing, and validating data, wherein said specific hardware and software comprise:
means for defining at least a first semantic model and a second semantic model;
means for defining a model mapping among semantic models;
means for storing said semantic models and said model mapping;
means for defining validation rules and transformation rules;
means for accepting data from at least one source;
means for transforming said data according to the model mapping;
means for validating said data; and,
means for forwarding said data to at least one destination.
