# US 11,308,130 B1 · Constructing ground truth when classifying data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,130 B1
- Document ID: US11308130B1
- Status: Granted
- Kind: B1
- Application number: 16/678,841
- Filed: 2019-11-08
- Granted: 2022-04-19
- Earliest claimed: 2017-10-11
- Assignees: Amperity, Inc.
- Applicants: Amperity, Inc.
- Inventors: Yan Yan; Stephen Meyles; Mona Akmal; Michael P. Fikes
- Web page: https://patentgrep.com/patent/US11308130B1

## CPC

- G06F 16/245: Query processing
- G06F 16/285: {Clustering or classification}
- G06F 16/353: into predefined classes
- G06N 20/00: Machine learning
- G06F 16/2228: {Indexing structures}
- G06F 16/355: Creation or modification of classes or clusters
- G06N 7/02: using fuzzy logic (computing arrangements based on biological models G06N3/00; computing arrangements using knowledge-based models G06N5/00)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure relates to evaluating whether two data records reflect the same entity using a classifier in the absence of ground truth. Without ground truth, it is difficult to determine the precision or recall of a classifier. The present disclosure generates a list comprising a series of unique feature signatures and a set of sample record pairs for each unique feature signature. In some embodiments, users may provide labels for the set of sample record pairs for each unique feature signature.

## Claims

### Claim 1

A system comprising:
a database that stores a plurality of records; and
a memory coupled to a processor, the memory comprising a plurality of instructions that cause the processor to:
perform pairwise comparisons on at least a portion of the plurality of records to generate feature signatures for the pairwise comparisons, wherein a given feature signature indicates common features between a pair of records being compared;
classify the feature signatures using a classifier; and
generate a list comprising a series of unique feature signatures and a set of sample record pairs for each of the unique feature signatures.

### Claim 2 (depends on 1)

The system of claim 1, wherein the plurality of instructions further cause the processor to perform a plurality of blocking functions to select pairs that are likely to be classified as related pairs.

### Claim 3 (depends on 1)

The system of claim 1, wherein the list further comprises respective frequencies that each unique feature signature occurs among the feature signatures.

### Claim 4 (depends on 3)

The system of claim 3, wherein the plurality of instructions further cause the processor to weight each feature signature according to its corresponding frequency.

### Claim 5 (depends on 1)

The system of claim 1, wherein the plurality of instructions further cause the processor to obtain user data for determining a precision value or recall value of the classifier.

### Claim 6 (depends on 5)

The system of claim 5, wherein the user data comprises labels for the set of sample record pairs for each of the unique feature signatures.

### Claim 7 (depends on 1)

The system of claim 1, wherein the plurality of instructions that cause the processor to perform pairwise comparisons further comprises instructions that cause the processor to determine one or more exact matches between features of a pair of records being compared.

### Claim 8

A system comprising:
a memory that stores a plurality of instructions;
a database that stores a plurality of records; and
a processor in data communication with the memory, the processor being configured to:
perform pairwise comparisons on at least a portion of the plurality of records to generate feature signatures for the pairwise comparisons, wherein a given feature signature indicates a degree of similarity of features between a pair of records being compared;
classify the feature signatures using a classifier; and
generate a list comprising a series of unique feature signatures and a set of sample record pairs for each of the unique feature signatures.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processor is configured to perform a plurality of blocking functions to select record pairs that are likely classified as related record pairs.

### Claim 10 (depends on 8)

The system of claim 8, wherein the classifier is a binary classifier.

### Claim 11 (depends on 8)

The system of claim 8, wherein the processor is configured to obtain user data comprising user-specified labels for the set of sample record pairs for each of the unique feature signatures.

### Claim 12 (depends on 11)

The system of claim 11, wherein the processor is configured to validate the classifier by determining a precision value or recall value using the user data.

### Claim 13 (depends on 12)

The system of claim 12, wherein the processor is configured to:
determine a frequency that each unique feature signature occurs by performing the pairwise comparisons on at least the portion of the plurality of records; and
weight each feature signature according to a corresponding frequency.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processor is configured to determine the precision value or recall value of the classifier based on a weighted feature signature and the user data.

### Claim 15

A computer implemented method comprising:
storing a plurality of records and a plurality of features for the plurality of records in a database;
performing pairwise comparisons on at least a portion of the plurality of records to generate feature signatures for each of the pairwise comparisons, wherein a given feature signature indicates a degree of similarity of features between a pair of records being compared;
classifying feature signatures using a classifier; and
generating a list comprising a series of unique feature signatures and a set of sample record pairs for each of the unique feature signatures.

### Claim 16 (depends on 15)

The method of claim 15, further comprising classifying each feature signature using a binary classifier.

### Claim 17 (depends on 15)

The method of claim 15, further comprising obtaining user data comprising user-specified labels for the set of sample record pairs for each unique feature signature.

### Claim 18 (depends on 17)

The method of claim 17, further comprising validating the classifier by determining a precision value or recall value using the user data.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
determining a frequency that each unique feature signature occurs by performing the pairwise comparisons on at least the portion of the plurality of records; and
weighting each feature signature according to a corresponding frequency.

### Claim 20 (depends on 19)

The method of claim 19, further comprising determining the precision value or recall value of the classifier based on a weighted feature signature and the user data.