Systems and methods for phrase clustering
AssigneeInternational Business Machines Corporation
InventorsIndrajit Bhattacharya, Shantanu Ravindra Godbole, Akshit Sharma
Life of this patent
Estimating the term…
Abstract
Systems and associated methods for enhanced concept understanding in large document collections through phrase clustering are described. Embodiments take as input an initial set of phrases and estimate centroids using a clustering process. Embodiments then generate new phrases around each of the current centroids using the current phrases. These new phrases are added to the current set, and the clustering process is iterated. Upon convergence, embodiments finalize clusters based on phrases of any given length.
Claims10
A computer program product for phrase based clustering comprising:
- a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
- computer readable program code configured to access a collection of items to be clustered;
- computer readable program code configured to receive an initial set of phrases as input;
- computer readable program code configured to cluster the collection of items to be clustered using the initial set of phrases to create centroids;
- computer readable program code configured to generate a new set of phrases around the centroids;
- computer readable program code configured to add the new set of phrases to the initial set of phrases to produce a combined set of phrases;
- and
- computer readable program code configured to re-cluster the collection of items to be clustered using the combined set of phrases;
- wherein to generate a new set of phrases around the centroids comprises:
- finding high weight words in a context vector for a centroid;
- finding existing phrases that appear around words of a centroid;
- and
- pruning phrases that do not have high weight for at least one of the words of the centroid;
- said pruning comprising:
- generating a higher-order phrase via combining two lower-order phrases, each of the higher-order phrase and the two lower-order phrases comprising a context vector;
- and
- employing a monotonicity property, wherein the higher-order phrase has high weight for a word in its context vector if both of the lower order phrases individually each have high weight for the at least one word in their context vectors.
A system for phrase based clustering comprising:
- one or more processors;
- and
- a memory operatively connected to the one or more processors;
- wherein, responsive to execution of computer readable program code accessible to the one or more processors, the one or more processors are configured to:
- access a collection of items to be clustered;
- receive an initial set of phrases as input;
- cluster the collection of items to be clustered using the initial set of phrases to create centroids;
- generate a new set of phrases around the centroids;
- add the new set of phrases to the initial set of phrases to produce a combined set of phrases;
- and
- re-cluster the collection of items to be clustered using the combined set of phrases;
- wherein to generate a new set of phrases around the centroids comprises:
- finding high weight words in a context vector for a centroid;
- finding existing phrases that appear around words of a centroid;
- and
- pruning phrases that do not have high weight for at least one of the words of the centroid;
- said pruning comprising:
- generating a higher-order phrase via combining two lower-order phrases, each of the higher-order phrase and the two lower-order phrases comprising a context vector;
- and
- employing a monotonicity property, wherein the higher-order phrase has high weight for a word in its context vector if both of the lower order phrases individually each have high weight for the at least one word in their context vectors.
Dependencies are inferred from the claim text.