Natural language processing with missing tokens in a corpus
AssigneeInternational Business Machines Corporation
InventorsRaj Nagesh, Charles Christopher Walker, Kriteshwar Kaur Kohli
Life of this patent
Estimating the term…
Abstract
Text blocks are semantically compared, and a semantic score is provided to a user. The semantic score is based on application of a machine learning model trained on a text corpus. One or both of the two text blocks may have one or more words that do not appear in the training text corpus (skip-words). Skip-words are used, rather than discarded, to adjust the semantic score via, for example, a penalization function. The user provides feedback about the accuracy of the adjusted semantic score, and the feedback is used to perform supervised learning model.
Claims19
A method for determining semantic similarity between two text blocks using an unsupervised machine learning model, comprising:
- accessing by a computer processor a text corpus, the text corpus stored in a database;
- training a neural network machine learning model with the accessed text corpus using an unsupervised learning process, the unsupervised learning process not requiring user feedback to train the neural network machine learning model;
- receiving two text blocks for comparison, the comparison performed using the trained unsupervised machine learning model;
- identifying an unobserved token in at least one of the two text blocks, wherein the unobserved token does not appear in the text corpus used to train the unsupervised machine learning model;
- and
- determining a semantic similarity between the two text blocks based on the unobserved token that does not appear in the text corpus used to train the machine learning model, the semantic similarity and the unobserved token used to retrain the machine learning model to improve accuracy.
A computer program product for determining semantic similarity between two text blocks using an unsupervised machine learning model, the computer program product comprising programming instructions embodied on one or more tangible storage media, the programming instructions being executable by one or more processors to perform a method, the programming instructions comprising instructions for:
- accessing by one or more processors a text corpus, the text corpus stored in a database;
- training a neural network machine learning model with the accessed text corpus using an unsupervised learning process, the unsupervised learning process not requiring user feedback to train the neural network machine learning model;
- receiving, by the one or more processors, two text blocks for comparison, the comparison performed using the trained unsupervised machine learning model;
- identifying, by the one or more processors, an unobserved token in at least one of the two text blocks, wherein the unobserved token does not appear in the text corpus used to train the unsupervised machine learning model;
- and
- determining, by the one or more processors, a semantic similarity between the two text blocks based on the unobserved token that does not appear in the text corpus used to train the machine learning model, the semantic similarity and the unobserved token used to retrain the machine learning model to improve accuracy.
A computer system for determining semantic similarity between two text blocks using an unsupervised machine learning model, comprising:
- one or more processors;
- and
- one or more tangible storage media storing programming instructions for execution by the one or more processors to perform a method, the programming instructions comprising instructions for:
- accessing a text corpus, the text corpus stored in a database;
- training a machine learning model with the text corpus using an unsupervised learning process, the unsupervised learning process not requiring user feedback to train the machine learning model;
- receiving two text blocks for comparison, the comparison performed using the trained unsupervised machine learning model;
- identifying an unobserved token in at least one of the two text blocks, wherein the unobserved token does not appear in the text corpus used to train the unsupervised machine learning model;
- and
- determining a semantic similarity between the two text blocks based on the unobserved token that does not appear in the text corpus used to train the machine learning model, the semantic similarity and the unobserved token used to retrain the machine learning model to improve accuracy.
Dependencies are inferred from the claim text.