# US 10,482,079 B2 · Data de-duplication systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,079 B2
- Document ID: US10482079B2
- Status: Granted
- Kind: B2
- Application number: 15/617,558
- Filed: 2017-06-08
- Published: 2017-09-21
- Granted: 2019-11-19
- Earliest claimed: 2014-11-20
- Assignees: CoreLogic Credco, LLC
- Applicants: CoreLogic Credco, LLC
- Inventors: Parag Vijay Ahire
- Web page: https://patentgrep.com/patent/US10482079B2

## CPC

- G06F 16/2365: {Ensuring data consistency and integrity}
- G06N 5/025: Extracting rules from data
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 40/03: Credit; Loans; Processing thereof
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A system, method, and computer program includes a communications interface configured to receive a set of industry reports from multiple industry sources, and circuitry to compare one or more attributes of at least two trade lines to identify whether the at least two trade lines are duplicates. The circuitry characterizes as a binary indication whether the comparing indicates the one or more attributes are a match, and display a representation of the binary indication and receive a user-identified indication whether the at least two trade lines are duplicates. The circuitry trains a classifier, records the indication whether the at least two trade lines are duplicates and removes at least one of the at least two trade lines from the set of industry reports, and runs the classifier. Subsequently, a supervised machine learning classifier is trained in fit on the training data and is evaluated for accuracy of the testing data.

## Claims

### Claim 1

A computer-implemented method of de-duplicating data, the computer-implemented method having steps comprising:
receiving data through a communications interface from a plurality of sources;
comparing with circuitry one or more attributes of at least two sets of data to identify whether the at least two sets of data are duplicates;
characterizing with the circuitry as a binary indication whether the comparing indicates the one or more attributes of the at least two sets of data are a match and storing the binary indication in a memory;
displaying on a display a representation of the binary indication and receiving via a graphical user interface (GUI) a user-identified indication whether the at least two sets of data are duplicates;
training a classifier through the circuitry with feedback from the user-identified indication, the classifier being a computer-implemented classifier;
recording in the memory the binary indication whether the at least two sets of data are duplicates and when the at least two sets of data are duplicates, removing at least one of the at least two sets of data so as to de-duplicate a portion of the at least two sets of data; and
running the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions, wherein the binary indication includes one or more of a true positive indicator, a true negative indicator, a false positive, and a false negative indicator, and an accuracy of the classifier is a function of a total sum of the true positive indicators and the true negative indicators, divided by a total sum of the true positive indicators, the false negative indicators, the false positive indicators and the true negative indicators.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
dividing data results of the comparing with other comparing steps to form a training set of data and a testing set of data.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the training set of data contains at least 60% of the data results and the testing set of data contains not more than 40% of the data results.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, further comprising:
applying the training set of data to fit the classifier via a k-fold cross-validation process; and
testing an accuracy of the classifier via the testing set of data.

### Claim 5 (depends on 2)

The computer-implemented method of claim 2, further comprising:
fitting with the circuitry a plurality of decision trees for an associated plurality of subsamples of the data results.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the fitting is executed via one of an Extra Trees Classifier, a Random Forest Classifier, or a Decision Tree Classifier implemented by the circuitry.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein a positive comparison of the one or more attributes of the at least two sets of data comprises blending of the data from the plurality of sources by triggering a de-duplication of the portion of the at least two sets of data.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein a negative comparison of the one or more attributes of the at least two sets of data comprises keeping the at least two sets of data as distinct sets of data without blending of the data from the plurality of sources.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the data includes a plurality of credit reports from different credit bureaus.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the data includes one or more reports from one of an electronic file archival report, an electronic indexing report, a census reporting report, an inventory recording report, a research organization report, a financial decision-making report, and an evaluation and achievement reporting report.

### Claim 11

A supervised machine learning de-duplication system, comprising:
a communications interface configured to receive data from a plurality of sources; and
processing circuitry configured to
compare one or more attributes of at least two sets of data to identify whether the at least two sets of data are duplicates,
characterize as a binary indication whether the comparing indicates the one or more attributes of the at least two sets of data are a match and store in a memory the binary indication
display on a display a representation of the binary indication and receive via a graphical user interface (GUI) a user-identified indication whether the at least two sets of data are duplicates,
train a classifier with feedback from the user-identified indication, the classifier being a computer-implemented classifier,
record in the memory the binary indication whether the at least two sets of data are duplicates and when the at least two sets of data are duplicates, remove at least one of the at least two sets of data so as to de-duplicate a portion of the at least two sets of data,
run the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions dividing data results of the comparing with other comparing steps to form a training set of data and a testing set of data;
apply the training set of data to fit the classifier via a k-fold cross-validation process;
test an accuracy of the classifier via the testing set of data; and
fit a plurality of decision trees for an associated plurality of sub-samples of the data results.

### Claim 12 (depends on 11)

The system of claim 11, wherein the processing circuitry is further configured to divide data results of the comparing with other comparing steps to form a training set of data and a testing set of data;
apply the training set of data to fit the classifier via a k-fold cross-validation process;
test an accuracy of the classifier via the testing set of data; and
fit a plurality of decision trees for an associated plurality of sub-samples of the data results.

### Claim 13 (depends on 11)

The system of claim 11, wherein a positive comparison of the one or more attributes of the at least two sets of data indicates a redundancy of at least one of the at least two sets of data and triggers a de-duplication of the portion of the at least two sets of data.

### Claim 14 (depends on 11)

The system of claim 11, wherein a negative comparison of the one or more attributes of the at least two sets of data indicates at most a partial redundancy and indicates the at least two sets of data should be kept as distinct sets of data without blending of the data from the plurality of sources.

### Claim 15 (depends on 11)

The system of claim 11, wherein the data includes a plurality of credit reports from different credit bureaus.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more attributes include one or more of an account number, an account type, a date of an account opening, a data repository source, a high credit amount, a lost or stolen indicator, a bureau identity, a subscriber information, a balance, a payment amount, and a credit limit.

### Claim 17 (depends on 11)

The system of claim 11, wherein the binary indication includes one or more of a true positive indicator, a true negative indicator, a false positive indicator, and a false negative indicator.

### Claim 18 (depends on 11)

The system of claim 11, wherein an accuracy of the classifier is a function of a total sum of the true positive indicators and the true negative indicators, divided by a total sum of the true positive indicators, the false negative indicators, the false positive indicators, and the true negative indicators.

### Claim 19

A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that when executed by circuitry performs a method of de-duplicating data, the method having steps comprising:
receiving data through a communications interface from a plurality of sources;
comparing one or more attributes of at least two sets of data to identify whether the at least two sets of data are duplicates;
characterizing as a binary indication whether the comparing indicates the one or more attributes of the at least two sets of data are a match and storing in a memory the binary indication;
displaying on a display a representation of the binary indication and receiving via a graphical user interface (GUI) a user-identified indication whether the at least two sets of data are duplicates;
training a classifier with feedback from the user-identified indication, the classifier being a computer-implemented classifier;
recording in the memory the binary indication whether the at least two sets of data are duplicates and when the at least two sets of data are duplicates, removing at least one of the at least two sets of data so as to de-duplicate a portion of the at least two sets of data; and
running the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions, wherein the binary indication includes one or more of a true positive indicator, a true negative indicator, a false positive, and a false negative indicator, and an accuracy of the classifier is a function of a total sum of the true positive indicators and the true negative indicators, divided by a total sum of the true positive indicators, the false negative indicators, the false positive indicators and the true negative indicators.