# US 2014/0195494 A1 · Method and system for creating and maintaining unique data repository

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0195494 A1
- Document ID: US20140195494A1
- Status: Published application
- Kind: A1
- Application number: 13/851,380
- Filed: 2013-03-27
- Published: 2014-07-10
- Earliest claimed: 2013-01-08
- Assignees: Tata Consultancy Services Limited
- Applicants: Tata Consultancy Services Limited
- Inventors: Nishant GURU; Amit PRAKASH; Arjun OJHA; Garima AIMA
- Web page: https://patentgrep.com/patent/US20140195494A1

## CPC

- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/16: File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F 16/174: Redundancy elimination performed by the file system (error detection or correction of the data by redundancy in operations G06F11/14)
- G06F 16/1748: {De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641)}
- G06F 16/1756: {based on delta files}
- G06F 16/2456: {Join operations}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In accordance with the disclosure, there is provided a system and method for creating and maintaining unique data repository comprising a matching process based on a set of predefined matching conditions and thereon performing an action type corresponding to the outcome of matching process. The present disclosure provides for real time data de-duplication and updation of unique data repository to obtain a unified view of unique and matching records.

## Claims

### Claim 1

A computer implemented method of managing and updating a data repository in real time, comprising:
retrieving at least one record from a stream of incoming records maintained in a load repository for matching the retrieved record with a previous record history maintained in a detail repository based on a first uniquely identifying matching key;
performing upon the retrieved record a sequence of matching runs executable on a parallel processing engine, said matching runs further comprising:
a) matching the incoming record against a set of records maintained along with their second associated unique identifier in a master repository, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;
b) for records found non matching in step a, iteratively performing a matching process between the non matching records and image records, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;
c) for records found non matching in steps a and b, iteratively performing a self join matching process between the records of the load repository, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result; and
d) for the records found non matching in steps a, b and c, identifying remaining similar matched records based on a criterion.

### Claim 2 (depends on 1)

The method of claim 1, wherein the detail repository is configured to maintain the history of all the incoming data record along with the first uniquely identifying matching key and the second unique identifier.

### Claim 3 (depends on 1)

The method of claim 1, wherein the matching conditions include at least one of:
a detail match that requires the first identifying matching keys of the incoming record and those of the detail repository to match;
a master match that requires the incoming records and those of the master repository to match on the basis of a batch criteria; and
a same match that requires each of the detail match and the master match condition to satisfy.

### Claim 4 (depends on 3)

The method of claim 3, wherein the action types include at least one of:
for the records satisfying the detail, the master and the same match conditions, performing an update action which updates records of the master and the detail repository with content of the incoming data set;
for the records satisfying the detail and the master conditions, performing an auto merge action which updates the records of the detail and the master repository with new content while marking previously linked master record as inactive if no other corresponding match exists for it in the detail repository;
for the records satisfying only the master match, performing an auto consolidate action which adds the incoming record as a new record in the detail repository while updating contents of the incoming record in the master repository;
for the records satisfying only the detail match, performing an update new action type characterized which adds the record in the master repository with a new value of the second unique identifier and thereupon updating for records matching the first uniquely identifiable matching key; and
for the records not satisfying either of the detail match and the master match, performing a new action type which adds the record as new record in the master repository and thereupon in the detail repository with new values of the second unique identifier and the first uniquely identifiable matching key respectively.

### Claim 5 (depends on 4)

The method of claim 4, wherein the action types determined for step a includes a combination of the Update, the auto consolidate and the auto merge action type.

### Claim 6 (depends on 4)

The method of claim 4, wherein the action types determined for step b includes a combination of the Update, the auto consolidate and the auto merge action type.

### Claim 7 (depends on 4)

The method of claim 4, wherein the action types determined for step c includes a combination of the auto consolidate, the auto merge, the update new and the new action type.

### Claim 8 (depends on 1)

The method of claim 1, wherein the step d further comprising:
for one or more unmatched records, inserting the corresponding image records, into the master record as a set of new records; and
for the matched records remaining after the steps a, b and c, applying a grouping technique to join all matching records as a group and assigning to each record in the group, a new group identification key.

### Claim 9

A system for managing and updating a unique data repository in real time, the system comprising:
a load repository configured to store a stream of records, each of the incoming record being associated with a first identifying matching key;
a parallel processing system for retrieving at least one record from the load repository and performing thereupon a sequence of matching runs and coordinating with a master repository that is configured to consolidate and store a set of matching records obtained from the matching run along with a corresponding second unique identifier, wherein said matching runs further include:
in a load-master run, matching the incoming record against a set of records maintained in the master repository, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;
in a load-image run, for records found non matching in the load master run, iteratively performing a matching process between the non matching records and image records, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;
in a load-load run, for records found non matching in the load-master and load-image run, iteratively performing a self join matching process between the records of the load repository, said matching based upon a set of predefined matching conditions and performing at least one suited action type predefined for each possible matching result; and
in an image-image run, for the records found non matching in the load-master run, load-image run, and load-load run, identifying for remaining similar matched records based on a criterion.

### Claim 10 (depends on 9)

The system of claim 9, wherein the matching conditions include at least one of:
a detail match which requires the first identifying matching keys of the incoming record and those of the detail repository to match;
a master match which requires the incoming records and those of the master repository to match on the basis of a batch criteria; and
a same match which requires each of the detail match and the master match condition to satisfy.

### Claim 11 (depends on 9)

The system of claim 9, wherein the action type includes at least one of:
for the records satisfying the detail, the master and the same match conditions, performing an update action which updates records of the master and the detail repository with content of the incoming data set;
for the records satisfying the detail and the master conditions, performing an auto merge action which updates the records of the master and the detail repository with new content while marking previously linked master record as inactive if no other corresponding match exists for it in the detail repository;
for the records satisfying only the master match, performing an auto consolidate action which adds the incoming record as a new record in the detail repository while updating contents of the incoming record in the master repository;
for the records satisfying only the detail match, performing an update new action type which adds the record in the master repository with a new value of the second unique identifier and thereupon updating for records matching the first uniquely identifiable matching key; and
for the records not satisfying either of the detail match and the master match, performing a new action type which adds the record as new record in the master repository and thereupon in the detail repository with new values of the second unique identifier and the first uniquely identifiable matching key respectively.