Intent classification using non-correlated features
AssigneeInternational Business Machines Corporation
InventorsAbhishek Shah, Ladislav Kunc, Haode Qi, Lin Pan, Saloni Potdar
Life of this patent
Estimating the term…
Abstract
A system for classifying a language sample intent by receiving a language sample including a set of features, identifying language sample features, determining a tokenization score for the language sample according to the language sample features, eliminating duplicate features according to the tokenization score, determining a term frequency (tf) according to the identified features and the tokenization score, determining an inverse document frequency (idf) according to the identified features and the tokenization score, and generating a term frequency-inverse document frequency (tf-idf) matrix for the identified features.
Claims20
A computer system comprising computer processors to implement a method for classifying a language sample, the computer system comprising:
- one or more computer processors;
- one or more computer readable storage devices;
- stored program instructions on one or more computer readable storage devices for execution by the one or more computer processors, the stored program instructions comprising:
- program instructions to train a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
- extracting language characteristics for each language sample,
- determining a probability of using white-space tokenizer (PWST) score for each language,
- training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
- program instructions to receive a first language sample comprising a set of features;
- program instructions to identify language sample features of the first language sample;
- program instructions to determine a tokenization score for the first language sample according to the language sample features;
- program instructions to determine a term frequency (tf) according to the identified language sample features and the tokenization score;
- program instructions to determine an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
- program instructions to generate a revised term frequency-inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf, and the idf;
- program instructions to receive input text;
- program instructions to identify tokens in the input text;
- and
- program instructions to translate the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.
A method for classifying a language sample, the method comprising:
- training, by one or more computer processors, a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
- extracting language characteristics for each language sample, determining a probability of using white-space tokenizer (PWST) score for each language,
- training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
- receiving, by the one or more computer processors, a first language sample comprising a set of features;
- identifying, by the one or more computer processors, language sample features of the first language sample;
- determining, by the one or more computer processors, a tokenization score for the first language sample according to the language sample features;
- determining, by the one or more computer processors, a term frequency (tf) according to the identified language sample features and the tokenization score;
- determining, by the one or more computer processors, an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
- generating, by the one or more computer processors, a revised term frequency—inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf and the idf;
- receiving, by the one or more computer processors, input text;
- identifying, by the one or more computer processors, tokens in the input text;
- and
- translating, by the one or more computer processors, the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.
A computer program product for classifying a language sample, the computer program product comprising one or more computer readable storage devices and collectively stored program instructions on the one or more computer readable storage devices, the stored program instructions comprising:
- program instructions to train a term frequency-inverse document frequency (tf-idf) matrix using a training data set including labeled language samples for multiple languages by:
- extracting language characteristics for each language sample,
- determining a probability of using white-space tokenizer (PWST) score for each language,
- training weights of a tf-idf matrix for each language according to the PWST and language characteristics for each language,
- program instructions to receive a first language sample comprising a set of features;
- program instructions to identify language sample features of the first language sample;
- program instructions to determine a tokenization score for the first language sample according to the language sample features;
- program instructions to determine a term frequency (tf) according to the identified language sample features and the tokenization score;
- program instructions to determine an inverse document frequency (idf) according to the identified language sample features and the tokenization score;
- program instructions to generate a revised term frequency—inverse document frequency (tf-idf) matrix for the identified language sample features using the trained tf-idf matrix, the tf, and the idf;
- program instructions to receive input text;
- program instructions to identify tokens in the input text;
- and
- program instructions to translate the tokens into entries in the revised term frequency—inverse document frequency (tf-idf) matrix to classify an intent of the input text.
Dependencies are inferred from the claim text.