System and method for decision driven hybrid text clustering

A method, comprising:

• receiving a plurality of text documents in a memory of a device comprising one or more processors;
• at the one or more processors:
• selecting a set of features in the plurality of text documents to represent a nature of the contents of the plurality of documents, each feature comprising an n-tuple ordered list of n elements in at least one of the text documents indicative of the nature of the contents of the plurality of text documents;
• assigning a respective weight to each feature in the set of features;
• computing a moment of each feature in the set of features, each moment based on a frequency of occurrence of the feature in the plurality of text documents and a probability of occurrence of each feature in each text document with respect to a length of the text document;
• computing a statistical coefficient index for each text document of the plurality of text documents, each statistical coefficient index for a text document obtained by:
• computing, for each feature in the text document, a product comprising the moment of the feature times the respective weight assigned to the feature, and
• computing a sum of each product corresponding to each feature in the text document to obtain the statistical coefficient index for the text document;
• computing a decision score for each text document of the plurality of text documents, each decision score comprising a sigmoid value of the statistical coefficient index of the text document, each sigmoid value in a range of 0 to 1;
• selecting a first threshold value between 0 and 1;
• splitting the plurality of text documents into a first group comprising text documents with a decision score less than or equal to the first threshold value and a second group comprising text documents with a decision score greater than the first threshold value;
• iteratively applying a clustering process to the text documents in the first group based on a distribution of the n-tuple features and the probability of occurrence of the n-tuple features in the text documents of the first group, wherein n=1, to determine an optimized number of clusters for the text documents in the first group and to assign the text documents in the first group to a given cluster of the first group based on a coherence score of semantically similar features in the text documents of the given cluster;
• iteratively applying the clustering process to the text documents in the second group based on a distribution of the n-tuple features and the probability of occurrence of the n-tuple features in the text documents of the second group, wherein n=2, to determine an optimized number of clusters for the text documents in the second group and to assign the text documents in the second group to a given cluster of the second group based on a coherence score of semantically similar features in the text documents of the given cluster;
• applying a count vectorization process to the features of each cluster of the first group to obtain a first set of feature vectors of the clusters of the first group;
• applying the count vectorization process to the features of each cluster of the second group to obtain a second set of feature vectors of the clusters of the second group;
• computing a semantic similarity score for pairs comprising a cluster of the first group and a cluster of the second group;
• merging the cluster of the first group with the cluster from the second group in the pair when the semantic similarity score is higher than a second threshold value to converge the clusters of the first group and the clusters of the second group into a final set of clusters of the plurality of text documents;
• and
• storing the final set of clusters of the plurality of text documents on a data storage device or displaying the final set of clusters on a user interface hardware.