Method for data and text mining and literature-based discovery

A method of determining levels of emphasis, comprising the steps of:

• selecting a collection of documents, each document containing at least one unstructured field;
• extracting all phrases from said unstructured field;
• filtering all extracted phrases to generate a list of high technical content phrases;
• generating a co-occurrence matrix of high technical content phrases for said unstructured field;
• normalizing matrix cell values of said co-occurrence matrix to generate a normalized matrix for said field;
• grouping phrases from said unstructured field by clustering techniques on said normalized matrix;
• summing the phrase frequencies of occurrence within each group, thereby indicating a level of emphasis for each group generated from said collection.

A method of generating a taxonomy of a collection of documents, comprising the steps of:

• selecting a collection of documents, each document containing at least one structured field;
• extracting all phrases from said structured field;
• factor matrix filtering all of said extracted phrases to generate a list of high technical content phrases;
• generating a co-occurrence matrix of said listed phrases for said field;
• normalizing cell values of said co-occurrence matrix to generate a normalized matrix for said field;
• grouping said listed phrases for each said field using clustering techniques on said normalized matrix.
• summing the frequencies of occurrence within each group, thereby indicating a level of emphasis for each group generated from said collection.

A method of literature-based problem solving, comprising the steps of:

• identifying a problem;
• selecting a source database comprising documents related to said problem, each of said documents including at least one unstructured field;
• retrieving all documents relevant to said problem from said source database to form a set of initially retrieved documents;
• extracting all phrases from said unstructured field of said set of initially retrieved documents;
• factor matrix filtering all of said extracted phrases to generate a first list of high technical content phrases;
• generating a co-occurrence matrix of said high technical content phrases from said first list;
• normalizing matrix cell values of said co-occurrence matrix to generate a normalized matrix for said field;
• grouping phrases from said unstructured field into thematic categories and subcategories by clustering techniques on said normalized matrix;
• generating a directly related topical literature for each said subcategory by retrieving documents related to each of said subcategories, each said directly related topical literature being disjoint with said selected documents and with said directly related topical literature from said other subcategories, each document in each said directly related topical literature including at least one unstructured field;
• extracting all phrases from said unstructured field of said directly related topical literature documents;
• filtering all of said extracted phrases from said directly related topical literature documents to generate a second list of high technical content phrases;
• generating a co-occurrence matrix of high technical content phrases from said second list;
• normalizing matrix cell values of said co-occurrence matrix to generate a normalized matrix for said unstructured field from said topical literature documents;
• grouping phrases from said unstructured field of said directly related topical literature documents into thematic categories by clustering techniques on said normalized matrix;
• dividing said thematic categories into a first set of categories representing specific solutions to said problem and a second set of categories that do not represent specific solutions to said problem;
• generating, for each of said second set of categories, a corresponding disjoint indirectly related literature;
• extracting all phrases from each said indirectly related literature;
• filtering all of said phrases extracted from said indirectly related literatures to generate a second list of high technical content phrases;
• grouping said high technical content phrases from said second list into thematic categories for each said indirectly related literature, the set of categories consisting of said first set of categories and said high technical content phrases from said second list into thematic categories for each said indirectly related literature to form a set of basis categories;
• generating, for each of said first set of categories, and for each of said indirectly related literature thematic categories that represent potential solutions to said problems, a third list of phrases, phrase combination, and phrase co-occurrences;
• filtering said third lists to remove all phrases and phrase combinations that appear in said initially retrieved documents, thereby forming filtered third lists;
• determining the number of categories and the sum of frequencies over all of said basis categories for each phrase and phrase co-occurrence on said filtered third list.
• ranking said phrases and phrase co-occurrences on said filtered third list by the number of categories and the sum of frequencies over all of said basis categories.