US 2024/0176795 A1Published application

Systems and methods of joining data records and detecting string similarity

AssigneeMastercard International Incorporated

InventorsTadeu Augusto FERREIRA, Ravi Santosh ARVAPALLY

Life of this patent

Estimating the term…

Loading patent…

Abstract

The disclosure relates to methods and systems of joining data structures based on a composite similarity score (CSS). For example, a computer system may use a plurality of similarity models to generate respective similarity scores. Each similarity score may be a metric that indicates a confidence that a first data value of a first data record is similar to a second data value of a second data record. The computer system may generate the CSS based on the plurality of similarity sub-scores. The CSS may indicate a confidence that the records being compared are similar. Thus, the CSS may be used to detect similar data records across different data structures. The disclosure also relates to a string similarity model that detects similarity among strings without respect to an order of words in each string and in a way that tolerates errors or omissions in one or both strings.

Claims20

Dependent claims

A system of joining data structures based on compound similarity between respective data records of the data structures, comprising:

a memory that stores a string similarity model that outputs a confidence metric that a pair of strings are similar to one another without respect to an order of words in either of the pair strings;
a processor programmed to:
generate, based on the string similarity model, a first string similarity score between a first string of a first data record in the first data structure and a second string of a second data record in the second data structure, wherein the first string similarity score is generated without respect to an order of a first plurality of segments in the first string and without respect to an order of a second plurality of segments in the second string;
generate, based on the string similarity model, a second string similarity score between a third string of the first data record and a fourth string of the second data record, wherein the second string similarity score is generated without respect to an order of a third plurality of segments in the third string and without respect to an order of a fourth plurality of segments in the fourth string;
determine a geocoordinate similarity score between first geocoordinate data in the first data record and second geocoordinate data in the second data record;
generate a compound similarity score based on the string similarity score, the second string similarity score, and the geocoordinate similarity score;
and
join the first data record and the second data record based on a determination that the compound similarity score exceeds a threshold compound similarity value.

5 more elements

A method of determining similarity between data, comprising:

determining, by a processor, an intersection between a first plurality of segments from a first data record and a second plurality of segments from a second data record, the intersection comprising exact matches between the first plurality of segments and the second plurality of segments;
determining, by the processor, a first difference set comprising any segments in the first plurality of segments that are not in the intersection;
determining, by the processor, a second difference set comprising any segments in the second plurality of segments that are not in the intersection;
for each first diff segment in the first difference set:
generating, by the processor, a segment similarity score between the first diff segment and each second diff segment;
determining, by the processor, a near-intersection based on a number of segment pairs in which the segment similarity score between the first diff segment and each second diff segment exceeds a segment similarity threshold value;
determining, by the processor, a non-intersection based on the first plurality of segments and the second plurality of segments, the non-intersection comprising segments that are non-exact matches and non-similar matches between the first plurality of segments and the second plurality of segments;
determining, by the processor, a difference in cardinality between the first difference set and the second difference set;
and
generating, by the processor, a confidence metric that the first data record is similar to the second data record based on the intersection, the near-intersection, the non-intersection, the difference in cardinality, and a parameter that controls a penalty imposed on the confidence metric based on the difference in cardinality.

7 more elements

A non-transitory computer readable medium storing instructions for determining data record matches between a first data record and a second data record, the first data record having a first plurality of words and the second data record having a second plurality of words, the instructions, when executed by a processor, causes the processor to:

determine an intersection between a first plurality of segments from a first data record and a second plurality of segments from a second data record, the intersection comprising exact matches between the first plurality of segments and the second plurality of segments;
determine a first difference set comprising any segments in the first plurality of segments that are not in the intersection;
determine a second difference set comprising any segments in the second plurality of segments that are not in the intersection;
for each first diff segment in the first difference set:
generate a segment similarity score between the first diff segment and each second diff segment;
determine a near-intersection based on a number of segment pairs in which the segment similarity score between the first diff segment and each second diff segment exceeds a segment similarity threshold value;
determine a non-intersection based on the first plurality of segments and the second plurality of segments, the non-intersection comprising segments that are non-exact matches and non-similar matches between the first plurality of segments and the second plurality of segments;
determine a difference in cardinality between the first difference set and the second difference set;
and
generate a confidence metric that the first data record is similar to the second data record based on the intersection, the near-intersection, the non-intersection, the difference in cardinality, and a parameter that controls a penalty imposed on the confidence metric based on the difference in cardinality.

7 more elements

Dependencies are inferred from the claim text.