Method and system for optimally searching a document database using a representative semantic space

A method for comparing documents for similarity by representative latent semantic analysis comprising:

• selecting documents based on a similarity criteria;
• generating a weighted term-by-document matrix for said selected documents;
• decomposing said weighted term-by-document matrix into a reduced concepts matrix, a term matrix, and a document matrix;
• discarding said document matrix;
• generating a first pseudo-document vector utilizing a first document, said reduced concepts matrix and said term matrix;
• generating a second pseudo-document vector utilizing a second document, said reduced concepts matrix and said term matrix;
• comparing said first pseudo-document vector to said second pseudo-document vector, and
• determining similarity of said first document and said second document based on the comparison of said first pseudo-document vector to said second pseudo-document vector.

A method for comparing documents for similarity by representative latent semantic analysis comprising:

• creating a concept database comprising:
• compiling a corpus from a plurality of documents representative of a similarity criteria;
• identifying terms occurring in the corpus;
• finding a global term weight for each identified term based on a frequency of occurrences in the corpus;
• finding a local term weight for each identified term based on a frequency of occurrences in each of the plurality of documents in the corpus;
• organizing said local term weights in a term-by-document matrix for the identified terms;
• creating a weighted term-by-document matrix by applying global term weights to corresponding local term weights for each of the respective identified terms in the term-by-document matrix;
• decomposing the weighted term-by-document matrix into at least a term matrix and a concepts matrix;
• forming a reduced concepts matrix by eliminating indices in said concept matrix identifying weak concepts;
• and
• forming a reduced term matrix by eliminating indices in said term matrix corresponding to the indices eliminated from the concept matrix;
• creating a document database from a collection of documents and said concept database comprising:
• compiling said collection of documents;
• generating a plurality of pseudo-document vectors utilizing said collection of documents and each of said reduced concepts matrix and said term matrix of said concept database;
• and
• determining document similarity between documents in the collection of documents comprising:
• comparing pseudo-document vectors from the document database for respective documents in the collection of documents;
• and
• determining similarity of said respective documents based on the comparison of the respective pseudo-document vectors from the document database.

A method for comparing documents for similarity by representative latent semantic analysis comprising:

• defining a representative semantic space;
• creating a first pseudo-document vector for a first document using at least a portion of the definition for the representative semantic space;
• creating a second pseudo-document vector for a second document using at least a portion of the definition for the representative semantic space;
• comparing the first pseudo-document vector to the second pseudo-document vector, and
• determining similarity of the first document and the second document based on the comparison of the first pseudo-document vector to the second pseudo-document vector.

A method for comparing resumès for similarity by representative latent semantic analysis comprising:

• generating a weighted term-by-resumè matrix for a group of resumès;
• decomposing said weighted term-by-resumè matrix into a term matrix of terms occurring in said group of resumès and a reduced concepts matrix of concepts, at least a portion of said concepts indicative of latent semantics in at least one of said resumès;
• identifying a collection of resumès, wherein at least one resumè in the collection of resumès is not included in said group of resumès;
• generating a plurality of pseudo-resumè vectors utilizing said collection of resumès and said reduced concepts matrix;
• generating a query pseudo-document vector utilizing a query document and said reduced concepts matrix;
• comparing said query pseudo-document vector to said plurality of pseudo-resumè vectors;
• and
• ranking said plurality of resumè based on similarity between said query pseudo-document vector to said plurality of pseudo-resumè vectors.

A method for comparing patent disclosures for similarity by representative latent semantic analysis comprising:

• generating a weighted term-by-patent matrix for a group of patent disclosures;
• decomposing said weighted term-by-patent disclosure matrix into a term matrix of terms occurring in said group of patent disclosures and a reduced concepts matrix of concepts, at least a portion of said concepts indicative of latent semantics in at least one of said patent disclosures;
• generating a plurality of pseudo-patent vectors utilizing said reduced concepts matrix and a plurality of patent disclosures;
• generating a query pseudo-document vector utilizing a query document and said reduced concepts matrix;
• comparing said query pseudo-document vector to said plurality of pseudo-patent disclosure vectors;
• and
• ranking said plurality of patent disclosures based on similarity between said query pseudo-document vector to said plurality of pseudo-patent vectors.

A system for comparing documents for similarity by representative latent semantic analysis comprising:

• a network containing a plurality of network elements;
• a server connected to said network, said server comprising:
• an interface for interfacing with the network;
• a memory, said memory containing instructions for:
• identifying terms occurring in the plurality of documents;
• determining a global term weight for each identified term;
• determining a local term weight for each identified term;
• organizing said local term weights in a term-by-document matrix for the identified terms;
• creating a weighted term-by-document matrix for global term weights and local term weight for each of the respective identified terms in the weighted term-by-document matrix;
• decomposing the weighted term-by-document matrix into at least a term matrix and a reduced concepts matrix;
• generating a pseudo-document vector utilizing a document, said reduced concepts matrix, and said term matrix;
• comparing pseudo-document vectors;
• and
• determining similarity of respective documents based on the comparison of their respective pseudo-document vectors;
• and
• a processor for executing instructions from memory.

A search engine method comparing documents for similarity by representative latent semantic analysis comprising:

• receiving a corpus of documents;
• creating a concept database from said corpus of documents, said concept database including at least a term matrix and a concepts matrix;
• receiving a collection of documents, wherein at least one document in said collection of documents is not in said corpus of documents;
• creating a document database from said collection of documents and said concept database, said documents database including at least one pseudo-document vector for each of at least two documents in said collection of documents, said at least one pseudo-document vector being created utilizing said each of at least two documents in said collection of documents and each of said reduced concepts matrix and said term matrix of said concept database;
• and
• determining document similarity between at least said each of at least two documents in said collection of documents using said pseudo-document vectors in said document database.

A data processing system implemented program product for performing a method for comparing documents for similarity by representative latent semantic analysis, the program product comprising:

• instructions for generating a weighted tern-by-document matrix for a group of selected documents, said group of selected documents being representative of a similarity criteria;
• instructions for decomposing said weighted term-by-document matrix into a term matrix of terms occurring said group of selected documents and a reduced concepts matrix of concepts, at least a portion of said concepts indicative of latent semantics in at least one of said selected documents;
• instructions for generating a first pseudo-document vector utilizing a first document and said reduced concepts matrix;
• instructions for generating a second pseudo-document vector utilizing a second document and said reduced concepts matrix;
• comparing said first pseudo-document vector to said second pseudo-document vector;
• and
• instructions for determining similarity of said first document and said second document based on the comparison of said first pseudo-document vector to said second pseudo-document vector.