Back to results
US 12,755,188 B2Granted
Entity resolution incorporating data from various data sources which uses tokens and normalizes records
AssigneeMicrosoft Technology Licensing, LLC
InventorsSatish J. Thomas, Murtaza Muidul Huda Chowdhury
Life of this patent
Estimating the term…
Abstract
A pair of records is tokenized to form a normalized representation of an entity represented by each record. The tokens are correlated to a machine learning system by determining whether a learned resolution already exists for the two entities. If not, the normalized records are compared to generate a comparison measure to determine whether the records match. The normalized records can also be used to perform a web search and web search results can be normalized and used as additional records for matching. When a match is found, the records are updated to indicate that they match, and the match is provided to the machine learning system to update the learned resolutions.
Claims20
A computer-implemented method performed by a computing system comprising one or more processors, memory, a partition component, a tokenization component, a vector generator, a threshold component, and a weighting component, the method comprising:
- obtaining, by the computing system, a plurality of records from different data sources, each record including an item identifier that identifies an item that is described by the record and an attribute that relates to the item;
- partitioning, by the partition component, the plurality of records into a plurality of record blocks based on partition criteria including at least attribute similarity;
- for each record block, processing, by the tokenization component, attribute data of each record to generate normalized entity tokens corresponding to the attributes of the record;
- selecting, within a record block, a first record and a second record;
- generating, by the vector generator, a similarity vector comprising a plurality of attribute-level similarity scores based on the entity tokens of the first and second records, wherein the similarity vector comprises attribute-level similarity scores generated across a plurality of attributes of the records;
- determining, by a threshold component, whether a similarity measure derived from the similarity vector satisfies a threshold value;
- responsive to the similarity measure not satisfying the threshold value:
- applying, by the weighting component, machine-learned attribute weights to generate a weighted similarity measure;
- and
- determining the match result based on at least one of the similarity measure or the weighted similarity measure;
- responsive to the match result indicating that the first and second records do not describe the same item;
- transmitting, by the computing system, at least one of the entity tokens generated from the normalized representation of at least one of the first and second records of a particular record block to a search engine as a search input;
- receiving search results;
- incorporating the search results into the particular record block from which the entity tokens were generated;
- repeating the generating of the similarity vector, determining whether the similarity measure satisfies the threshold value, and determining the match result using the particular record block including the incorporated search results to determine an updated match result based on the incorporated search results;
- and
- determining, by the computing system, whether the first and second records correspond to a same entity based on the updated match result.
14 more elements
A computing system comprising:
- at least one processor;
- and
- memory storing instructions executable by the at least one processor, wherein the instructions, when executed, cause the computing system to:
- obtain a plurality of records from different data sources, each record including an item identifier that identifies an item that is described by the record and an attribute that relates to the item;
- partition, by a partition component, the plurality of records into a plurality of record blocks based on partition criteria including at least attribute similarity;
- for each record block, process, by a tokenization component, attribute data of each record to generate normalized entity tokens corresponding to the attributes of the record;
- select, within a record block, a first record and a second record;
- generate, by a vector generator, a similarity vector comprising a plurality of attribute-level similarity scores based on the entity tokens of the first and second records, wherein the similarity vector comprises attribute-level similarity scores generated across a plurality of attributes of the records;
- determine, by a threshold component, whether a similarity measure derived from the similarity vector satisfies a threshold value;
- responsive to the similarity measure not satisfying the threshold value:
- apply, by a weighting component, machine-learned attribute weights to generate a weighted similarity measure;
- and
- determine the match result based on at least one of:
- the similarity measure or the weighted similarity measure;
- responsive to the match result indicating that the first and second records do not describe the same item;
- transmit, by the computing system, at least one of the entity tokens generated from the normalized representation of at least one of the first and second records of a particular record block to a search engine as a search input;
- receive search results;
- incorporate the search results into the particular record block from which the entity tokens were generated;
- and
- repeat generation of the similarity vector, determination of whether the similarity measure satisfies the threshold value, and determination of the match result using the particular record block including the incorporated search results to determine an updated match result based on the incorporated search results;
- and
- determine, by the computing system, whether the first and second records correspond to a same entity based on the updated match result.
19 more elements
An entity resolution system comprising:
- a partition component configured to:
- receive an input record set that includes records from a plurality of different data sources each record relating to an entity;
- and
- partition the input record set into a plurality of record blocks based on partition criteria, including at least one of geographic location or attribute similarity;
- and
- an entity matching component configured to:
- select first and second records from a record block;
- process attribute data of the first and second records to generate normalized entity tokens corresponding to attributes of the first and second records;
- generate a similarity vector comprising a plurality of attribute-level similarity scores based on the normalized entity tokens of the first and second records;
- determine whether a similarity measure derived from the similarity vector satisfies a threshold value;
- responsive to the similarity measure not satisfying the threshold value:
- apply machine-learned attribute weights to generate a weighted similarity measure;
- and
- determine a match result indicative of whether the first and second records resolve to a same entity based on the similarity measure or the weighted similarity measure;
- and
- a search engine interface configured to:
- transmit at least one of the normalized entity tokens corresponding to at least one of the first and second records to a search engine in response to the match result indicating that the first and second records do not resolve to the same entity;
- and
- receive search results corresponding to the transmitted normalized entity tokens;
- wherein the entity matching component is further configured to:
- incorporate at least some of the search results into the same record block;
- perform subsequent matching operations on the record block including the incorporated search results by repeating generation of the similarity vector, determining whether the similarity measure satisfies the threshold value, and determining an updated match result based on the incorporated search results;
- and
- determine whether the first and second records correspond to a same entity based on the updated match result.
21 more elements
Dependencies are inferred from the claim text.