System and method for performing efficient document scoring and clustering

A computer-implemented method for providing efficient document scoring of concepts within and clustering of documents in an electronically-stored document set, comprising:

• scoring a document in an electronically-stored document set, comprising:
• determining a frequency of occurrence of at least one concept within a document;
• analyzing a concept weight reflecting a specificity of meaning for the at least one concept within the document, wherein the concept weight is based on a number of terms for the at least one concept;
• analyzing a structural weight reflecting a degree of significance based on structural location within the document for the at least one concept;
• analyzing a corpus weight inversely weighing a reference count of occurrences for the at least one concept within the document;
• and
• evaluating a score to be associated with the at least one concept as a function of a summation of the frequency, concept weight, structural weight, and corpus weight and in accordance with the formula:
• S
• i
• =
• ∑
• 1
• ->
• n
• j
• ⁢
• f
• ij
• ×
• cw
• ij
• ×
• sw
• ij
• ×
• rw
• ij
• where Si comprises the score, fij comprises the frequency, 0<cwij≦1 comprises the concept weight, 0<swij≦1 comprises the structural weight, and 0<rwij≦1 comprises the corpus weight for occurrence j of concept i;
• forming the score assigned to the at least one concept as a normalized score vector for each such document in the electronically-stored document set;
• determining a similarity between the normalized score vector for each such document as an inner product of each normalized score vector;
• grouping the documents by the score into a plurality of clusters, comprising:
• selecting a set of candidate seed documents from the electronically-stored document set;
• identifying seed documents by applying the similarity to each such candidate seed document and selecting those candidate seed documents that are sufficiently unique from other candidate seed documents as the seed documents;
• identifying a plurality of non-seed documents;
• determining the similarity between each non-seed document and a center of each cluster;
• and
• assigning each non-seed document to the cluster with a best fit, subject to a minimum fit;
• and
• relocating outlier documents, comprising:
• determining the similarity between each of the documents grouped into each cluster based on the center of the cluster and the scores assigned to each of the at least one concepts in that document;
• dynamically determining a threshold for each cluster as a function of the similarity between each of the documents;
• and
• identifying and reassigning each of the documents with the similarity falling outside the threshold.

A computer-readable storage medium holding code for providing efficient document scoring of concepts within and clustering of documents in an electronically-stored document set, comprising:

• code for scoring a document in an electronically-stored document set, comprising:
• code for determining a frequency of occurrence of at least one concept within a document;
• code for analyzing a concept weight reflecting a specificity of meaning for the at least one concept within the document, wherein the concept weight is based on a number of terms for the at least one concept;
• code for analyzing a structural weight reflecting a degree of significance based on structural location within the document for the at least one concept;
• code for analyzing a corpus weight inversely weighing a reference count of occurrences for the at least one concept within the document;
• and
• code for evaluating a score to be associated with the at least one concept as a function of a summation of the frequency, concept weight, structural weight, and corpus weight in accordance with the formula:
• S
• i
• =
• ∑
• 1
• ->
• n
• j
• ⁢
• f
• ij
• ×
• cw
• ij
• ×
• sw
• ij
• ×
• rw
• ij
• where Si comprises the score, fij comprises the frequency, 0<cwij≦1 comprises the concept weight, 0<swij≦1 comprises the structural weight, and 0<rwij≦1 comprises the corpus weight for occurrence j of concept i;
• code for forming the score assigned to the at least one concept as a normalized score vector for each such document in the electronically-stored document set;
• code for determining a similarity between the normalized score vector for each such document as an inner product of each normalized score vector;
• code for grouping the documents by the score into a plurality of clusters, comprising:
• code for selecting a set of candidate seed documents from the electronically-stored document set;
• code for identifying seed documents by applying the similarity to each such candidate seed document and selecting those candidate seed documents that are sufficiently unique from other candidate seed documents as the seed documents;
• code for identifying a plurality of non-seed documents;
• code for determining the similarity between each non-seed document and a center of each cluster;
• and
• code for assigning each non-seed document to the cluster with a best fit, subject to a minimum fit;
• and
• code for relocating outlier documents, comprising:
• code for determining the similarity between each of the documents grouped into each cluster based on the center of the cluster and the scores assigned to each of the at least one concepts in that document;
• code for dynamically determining a threshold for each cluster as a function of the similarity between each of the documents;
• and
• code for identifying and reassigning each of the documents with the similarity falling outside the threshold.

An apparatus for providing efficient document scoring of concepts within and clustering of documents in an electronically-stored document set, comprising:

• means for scoring a document in an electronically-stored document set, comprising:
• means for determining a frequency of occurrence of at least one concept within a document;
• means for analyzing a concept weight reflecting a specificity of meaning for the at least one concept within the document, wherein the concept weight is based on a number of terms for the at least one concept;
• means for analyzing a structural weight reflecting a degree of significance based on structural location within the document for the at least one concept;
• means for analyzing a corpus weight inversely weighing a reference count of occurrences for the at least one concept within the document;
• and
• means for evaluating a score to be associated with the at least one concept as a function of a summation of the frequency, concept weight, structural weight, and corpus weight in accordance with the formula:
• S
• i
• =
• ∑
• 1
• ->
• n
• j
• ⁢
• f
• ij
• ×
• cw
• ij
• ×
• sw
• ij
• ×
• rw
• ij
• where Si comprises the score, fij comprises the frequency, 0<cwij≦1 comprises the concept weight, 0<swij≦1 comprises the structural weight, and 0<rwij≦1 comprises the corpus weight for occurrence j of concept i;
• means for forming the score assigned to the at least one concept as a normalized score vector for each such document in the electronically-stored document set;
• means for determining a similarity between the normalized score vector for each such document as an inner product of each normalized score vector;
• means for grouping the documents by the score into a plurality of clusters, comprising:
• means for selecting a set of candidate seed documents from the electronically-stored document set;
• means for identifying seed documents by applying the similarity to each such candidate seed document and selecting those candidate seed documents that are sufficiently unique from other candidate seed documents as the seed documents;
• means for identifying a plurality of non-seed documents;
• means for determining the similarity between each non-seed document and a center of each cluster;
• and
• means for assigning each non-seed document to the cluster with a best fit, subject to a minimum fit;
• and
• means for relocating outlier documents, comprising:
• means for determining the similarity between each of the documents grouped into each cluster based on the center of the cluster and the scores assigned to each of the at least one concepts in that document;
• means for dynamically determining a threshold for each cluster as a function of the similarity between each of the documents;
• and
• means for identifying and reassigning each of the documents with the similarity falling outside the threshold.