Methods and systems for word edit distance embedding
AssigneeCollibra Belgium BV
InventorsMichael Tandecki, Michael Maes, Anna Filipiak
Life of this patent
Estimating the term…
Abstract
A system for classifying words in a batch of words can include at least one memory device storing instructions for causing at least one processor to create dictionary vectors for each of a plurality of dictionary words using a neural network (NN), store each dictionary vector along with a classification indicator corresponding to the associated dictionary word, and create word vectors for each word in a batch of words for classification using the NN. The closest matching dictionary vectors are found for each word vector and the classification indicators of the closest matching dictionary vector for each word vector in the batch is reported.
Claims20
A system for classifying words in a batch of words, comprising:
- at least one memory device storing instructions for causing at least one processor to:
- create dictionary vectors for each of a plurality of dictionary words using a neural network (NN);
- determine a similarity metric for multiple pairs of training words;
- create a pair of training vectors for each pair of the training words using the NN and a twin of the NN;
- determine a cosine similarity between each pair of training vectors;
- train the NN to learn a transformation of each training word using each pair of training vectors, wherein the cosine similarity between each pair of training vectors is equivalent to the similarity metric for the multiple pairs of training words;
- create word vectors for each word in a batch of words for classification using the NN;
- find one or more closest matching dictionary vectors for each word vector;
- and
- assign a similarity value to each word vector, wherein the similarity value indicates a similarity between each word vector and the one or more closest matching dictionary vectors.
A system for classifying words in a batch of words, comprising:
- at least one memory device storing instructions for causing at least one processor to:
- train a convolutional neural network (CNN), including:
- encoding each of a plurality of training words into matrix form;
- calculating a similarity metric for multiple pairs of the plurality of training words;
- creating a pair of training vectors for each pair of encoded training words corresponding to the multiple pairs using the CNN and a twin of the CNN;
- calculating a cosine similarity between each pair of training vectors;
- comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the CNN based on comparing the similarity metric and the cosine similarity for each pair of training words;
- encode each of a plurality of dictionary words into matrix form;
- create dictionary vectors for each of the plurality of encoded dictionary words using the trained CNN;
- store each dictionary vector and a classification indicator corresponding to an associated dictionary word, wherein classification indicators include data identifying at least one of a first name, a last name, a country, and a minimum similarity threshold;
- encode each word in the batch of words for classification into matrix form;
- create word vectors for each encoded word in the batch using the trained CNN;
- find one or more closest matching dictionary vectors for each word vector;
- and
- assign a similarity value to each word vector, wherein the similarity value indicates a similarity between each word vector and the one or more closest matching dictionary vectors.
A system for classifying words in a batch of words, comprising:
- at least one memory device storing instructions for causing at least one processor to:
- train a neural network (NN), including:
- calculating a similarity metric for multiple pairs of a plurality of training words;
- creating a pair of training vectors for each of the multiple pairs of training words using the NN and a twin of the NN;
- calculating a cosine similarity between each pair of training vectors;
- and
- comparing the similarity metric and the cosine similarity for each pair of training words and adjusting the NN based on comparing the similarity metric and the cosine similarity,
- wherein the NN is trained to learn a transformation of each training word using each pair of training vectors, and
- wherein the cosine similarity between each pair of training vectors is equivalent to the similarity metric for the multiple pairs of training words.
Dependencies are inferred from the claim text.