System and method for efficiently generating cluster groupings in a multi-dimensional concept space

A method for building a multi-dimensional semantic concept space over a stored document collection, comprising:

• identifying a plurality of documents within a stored document collection containing substantially correlated terms reflecting syntactic content, comprising:
• extracting the terms in literal form from the documents;
• selecting the terms having frequencies of occurrence falling within a predefined threshold as being substantially correlated;
• generating a vector reflecting latent semantic similarities discovered between substantially correlated documents logically projected at an angle θ from a common axis in a concept space;
• forming one or more arbitrary clusters at an angle σ from the common axis in the concept space, each cluster comprising documents having such an angle θ falling within a predefined variance of the angle σ for the cluster;
• and
• constructing a new arbitrary cluster at an angle σ from the common axis in the concept space, each new cluster comprising documents having such an angle θ falling outside the predefined variance of the angle σ for the remaining clusters.

A system for efficiently generating cluster groupings in a multi-dimensional concept space, comprising:

• an extraction module extracting a plurality of terms from each document in a collection of stored unstructured documents, comprising:
• an extractor extracting the terms in literal form from the documents;
• a selector selecting the terms having frequencies of occurrence falling within a redefined threshold as being substantially correlated;
• and
• a cluster module building a concept space over the document collection, comprising:
• an identifier submodule identifying terms substantially correlated between a plurality of documents within the document collection;
• a mapping submodule expressing each correlated term as a vector mapped along an angle θ originating from a common axis in the concept space;
• a difference submodule determining a difference between the angle θ for each document and an angle σ for each cluster within the concept space;
• a build submodule populating an arbitrary cluster with those documents having such difference between the angle θ for each such document and the angle σ for each such cluster falling within a predetermined variance and creating a new arbitrary cluster within the concept space those documents having such difference between the angle θ for each such document and the angle σ for each such cluster falling outside the predetermined variance.

A method for efficiently generating cluster groupings in a multi-dimensional concept space, comprising:

• extracting a plurality of terms from each document in a collection of stored unstructured documents;
• and
• building a concept space over the document collection, comprising:
• identifying terms substantially correlated between a plurality of documents within the document collection, comprising:
• extracting the terms in literal form from the documents;
• selecting the terms having frequencies of occurrence falling within a predefined threshold as being substantially correlated;
• expressing each correlated term as a vector mapped along an angle θ originating from a common axis in the concept space;
• determining a difference between the angle θ for each document and an angle σ for each cluster within the concept space;
• populating an arbitrary cluster with those documents having such difference between the angle θ for each such document and the angle σ for each such cluster falling within a predetermined variance;
• and
• creating a new arbitrary cluster within the concept space those documents having such difference between the angle θ for each such document and the angle σ for each such cluster falling outside the predetermined variance.