Document information retrieval using global word co-occurrence patterns

A method, using a processor and memory, for determining relevant documents in a corpus of documents, the memory including a thesaurus of word vectors and a context vector for each document, the word vectors based on co-occurrence of words within each of the documents of the corpus of documents and each context vector based on the word vectors from the thesaurus for each word located in the corresponding document, the method comprising:

• inputting a query;
• generating a context vector for the query based on the word vectors from the thesaurus for each word in the query;
• determining a correlation coefficient for each document based on the context vectors for that document and the query;
• ranking each document based on the determined correlation coefficients;
• and
• outputting the ranking for at least one of the documents.

A method, using a processor and memory, for determining relevant documents in a corpus of documents, the memory including a thesaurus of word vectors and a context vector for each document, the word vectors based on co-occurrence of words within each of the documents of the corpus of documents and each context vector based on the word vectors from the thesaurus for each word located in the corresponding document, the method comprising:

• inputting a query;
• generating a factor vector for the query based on a clustering of the word vectors of the query;
• determining a correlation coefficient for each document based on the factor vector and the context vector for that document;
• ranking each document within a factor cluster based on the determined correlation coefficients;
• and
• determining a maximum rank of each document based on a combination of the ranks of the documents in each factor cluster;
• and
• outputting a final rank for at least one of the documents.

An apparatus for generating a thesaurus of word vectors for each word in a corpus of documents, the word vectors being based on the lexical co-occurrence of words within each of the documents, comprising:

• a memory containing the corpus of documents;
• an extractor for retrieving a word from the corpus;
• a counter recording the number of times the word co-occurs with a co-occurring word located within a predetermined range, the co-occurring word being any word located before and after the word within the predetermined range, the counter recording the number for every co-occurring word within the predetermined range;
• a generator generating a word vector for the word based on every recorded number;
• and
• an output for outputting the word vectors in the thesaurus.

An apparatus for retrieving relevant documents from a corpus of documents based on a query, comprising:

• a memory containing the corpus of documents;
• a thesaurus of word vectors for each word of the corpus, the word vectors being based on lexical co-occurrence of words within each of the documents;
• a processor for generating document vectors and a query vector based on the word vectors, each document vector being a summation of the word vectors that are associated with the words located within the document;
• the query vector being a summation of the word vectors that are associated with the words located within the query;
• determining means for determining a co-occurrence correlation relationship between the document vectors and the query vector;
• and
• output means for outputting the relevant documents based on the correlation relationship determined by the determining means.