# US 11,308,133 B2 · Entity matching using visual information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,133 B2 - Document ID: US11308133B2 - Status: Granted - Kind: B2 - Application number: 16/147,126 - Filed: 2018-09-28 - Published: 2020-04-02 - Granted: 2022-04-19 - Earliest claimed: 2018-09-28 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Sunhwan Lee; Pawan Chowdhary; Guangjie Ren - Web page: https://patentgrep.com/patent/US11308133B2 ## CPC - G06F 16/288: {Entity relationship models} - G06F 16/26: Visual data mining; Browsing structured data - G06F 16/285: {Clustering or classification} - G06F 16/313: {Selection or weighting of terms for indexing} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract One embodiment provides a method comprising receiving a first set of information corresponding to a first entity and receiving a second set of information corresponding to a second entity. Each set of information received corresponding to each entity comprises visual information relating to the entity. The method further comprises encoding each set of information, and classifying the first entity and the second entity as either matching entities or non-matching entities based on the encoding. ## Claims ### Claim 1 A method comprising: receiving a first set of information corresponding to a first entity, wherein the first set of information comprises first text information relating to the first entity and further comprises one or more first images of the first entity; receiving a second set of information corresponding to a second entity, wherein the second set of information comprises second text information relating to the second entity and further comprises one or more second images of the second entity; performing visual encoding by: generating a first set of visual embeddings by embedding the first images of the first entity using a convolutional neural network (CNN); generating a second set of visual embeddings by embedding the second images of the second entity using the CNN; and encoding the first set of visual embeddings and the second set of visual embeddings to a first set of statistics indicative of a similarity between the first images of the first entity and the second images of the second entity by applying a pairwise comparison algorithm to the first set of visual embeddings and the second set of visual embeddings; performing text encoding by: encoding the first text information relating to the first entity to a first set of features indicative of occurrence of terms within the first text information; and encoding the second text information relating to the second entity to a second set of features indicative of occurrence of terms within the second text information; and classifying the first entity and the second entity as either matching entities or non-matching entities based on a concatenation of the first set of statistics, the first set of features, and the second set of features. ### Claim 2 (depends on 1) The method of claim 1, wherein each set of information corresponding to each entity includes one of a plurality of images of the entity or a single image of the entity, and each set of visual embeddings comprises a set of embedded images. ### Claim 3 (depends on 1) The method of claim 1, wherein the first set of statistics comprise one of the following: a maximum cosine similarity between the first set of visual embeddings and the second set of visual embeddings, a top n cosine similarity between the first set of visual embeddings and the second set of visual embeddings, or a top n similar visual embeddings between the first set of visual embeddings and the second set of visual embeddings. ### Claim 4 (depends on 1) The method of claim 1, wherein classifying the first entity and the second entity as either matching entities or non-matching entities based on a concatenation of the first set of statistics, the first set of features, and the second set of features comprises: utilizing a classifier trained using a supervised learning algorithm, wherein the classifier is configured to classify the first entity and the second entity as either matching entities or non-matching entities based in part on the first set of statistics. ### Claim 5 (depends on 4) The method of claim 4, wherein the first set of information further comprises first numeric information relating to the first entity, and the second set of information further comprises second numeric information relating to the second entity. ### Claim 6 (depends on 4) The method of claim 4, wherein the classifier is configured to classify the first entity and the second entity as either matching entities or non-matching entities further based in part on the first set of features and the second set of features. ### Claim 7 (depends on 6) The method of claim 6, wherein each set of features comprises one of the following statistics or models: term frequency-inverse document frequency (tf-idf), bag-of-words, or paragraph2vec. ### Claim 8 (depends on 4) The method of claim 4, wherein the classifier is further configured to classify the first entity and the second entity as either matching entities or non-matching entities further based in part on a second set of statistics. ### Claim 9 (depends on 8) The method of claim 8, wherein the second set of statistics comprise one of the following: a Jaccard distance between the first text information relating to the first entity and the second text information relating to the second entity, or a cosine similarity between the first text information relating to the first entity and the second text information relating to the second entity. ### Claim 10 (depends on 5) The method of claim 5, further comprising: performing numeric encoding by encoding the first numeric information relating to the first entity and the second numeric information relating to the second entity to a second set of statistics indicative of a similarity between the first numeric information relating to the first entity and the second numeric information relating to the second entity, wherein the classifier is further configured to classify the first entity and the second entity as either matching entities or non-matching entities further based in part on the second set of statistics. ### Claim 11 (depends on 10) The method of claim 10, wherein the second set of statistics comprise one of the following: a difference between the first numeric information relating to the first entity and the second numeric information relating to the second entity, or a cosine similarity between the first numeric information relating to the first entity and the second numeric information relating to the second entity. ### Claim 12 (depends on 4) The method of claim 4, wherein the classifier is one of a neural network, a support vector machine, or a decision tree. ### Claim 13 A system comprising: at least one processor; and a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including: receiving a first set of information corresponding to a first entity, wherein the first set of information comprises first text information relating to the first entity and further comprises one or more first images of the first entity; receiving a second set of information corresponding to a second entity, wherein the second set of information comprises second text information relating to the second entity and further comprises one or more second images of the second entity; performing visual encoding by: generating a first set of visual embeddings by embedding the first images of the first entity using a convolutional neural network (CNN); generating a second set of visual embeddings by embedding the second images of the second entity using the CNN; and encoding the first set of visual embeddings and the second set of visual embeddings to a first set of statistics indicative of a similarity between the first images of the first entity and the second images of the second entity by applying a pairwise comparison algorithm to the first set of visual embeddings and the second set of visual embeddings; performing text encoding by: encoding the first text information relating to the first entity to a first set of features indicative of occurrence of terms within the first text information; and encoding the second text information relating to the second entity to a second set of features indicative of occurrence of terms within the second text information; and classifying the first entity and the second entity as either matching entities or non-matching entities based on a concatenation of the first set of statistics, the first set of features, and the second set of features. ### Claim 14 (depends on 13) The system of claim 13, wherein classifying the first entity and the second entity as either matching entities or non-matching entities based on a concatenation of the first set of statistics, the first set of features, and the second set of features comprises: utilizing a classifier trained using a supervised learning algorithm, wherein the classifier is configured to classify the first entity and the second entity as either matching entities or non-matching entities based in part on the first set of statistics. ### Claim 15 (depends on 14) The system of claim 14, wherein the classifier is configured to classify the first entity and the second entity as either matching entities or non-matching entities further based in part on the first set of features and the second set of features. ### Claim 16 (depends on 14) The system of claim 14, wherein the classifier is further configured to classify the first entity and the second entity as either matching entities or non-matching entities further based in part on a second set of statistics. ### Claim 17 (depends on 14) The system of claim 14, wherein the operations further comprise: performing numeric encoding by encoding the first numeric information relating to the first entity and the second numeric information relating to the second entity to a second set of statistics indicative of a similarity between the first numeric information relating to the first entity and the second numeric information relating to the second entity, wherein the classifier is further configured to classify the first entity and the second entity as either matching entities or non-matching entities further based in part on the second set of statistics. ### Claim 18 A computer program product comprising a computer-readable hardware storage medium having program code embodied therewith, the program code being executable by a computer to implement a method comprising: receiving a first set of information corresponding to a first entity, wherein the first set of information comprises first text information relating to the first entity and further comprises one or more first images of the first entity; receiving a second set of information corresponding to a second entity, wherein the second set of information comprises second text information relating to the second entity and further comprises one or more second images of the second entity; performing visual encoding by: generating a first set of visual embeddings by embedding the first images of the first entity using a convolutional neural network (CNN); generating a second set of visual embeddings by embedding the second images of the second entity using the CNN; and encoding the first set of visual embeddings and the second set of visual embeddings to a first set of statistics indicative of a similarity between the first images of the first entity and the second images of the second entity by applying a pairwise comparison algorithm to the first set of visual embeddings and the second set of visual embeddings; performing text encoding by: encoding the first text information relating to the first entity to a first set of features indicative of occurrence of terms within the first text information; and encoding the second text information relating to the second entity to a second set of features indicative of occurrence of terms within the second text information; and classifying the first entity and the second entity as either matching entities or non-matching entities based on a concatenation of the first set of statistics, the first set of features, and the second set of features.