Multi-Stage Query Processing System and Method for Use with Tokenspace Repository

A multi-stage query processing system, comprising:

• one or more processors;
• memory;
• and
• one or more programs stored in the memory, the on or more programs comprising instructions for:
• performing a first stage processing of a query, including:
• retrieving a first set of document identifiers from an index in response to one or more query terms;
• generating a first set of relevancy scores for a first set of compressed documents corresponding to at least a subset of the first set of document identifiers based on one or more of:
• presence of query terms, term frequency, and document popularity;
• and
• storing the first set of relevancy scores in the memory;
• performing a second stage processing of the query, including:
• generating a second set of relevancy scores for the documents in the first set of compressed documents based on one or more of:
• a list of token positions for one or more query terms in the query, distances between query terms in the documents, attributes of tokens in the documents, and text that appears around a query term used in a document of the first set of documents;
• and
• storing the second set of relevancy scores in the memory;
• reading the first and second set of relevancy scores from the memory, and generating an ordered list of documents for further processing based on the first and second set of relevancy scores;
• automatically generating additional query terms from the documents in the ordered list of documents;
• formulating a new query using the additional query terms;
• processing the new query to retrieve a second set of document identifiers from the index and to generate a third set of relevancy scores based at least in part on the additional query terms;
• and
• using the third set of relevancy scores to select a set of top documents for presentation to the user.

A non-transitory computer-readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

• performing a first stage processing of a query, including:
• retrieving a first set of document identifiers from an index in response to one or more query terms;
• generating a first set of relevancy scores for a first set of compressed documents corresponding to at least a subset of the first set of document identifiers based on one or more of:
• presence of query terms, term frequency, and document popularity;
• and
• storing the first set of relevancy scores in the memory;
• performing a second stage processing of the query, including:
• generating a second set of relevancy scores for the documents in the first set of compressed documents based on one or more of:
• a list of token positions for one or more query terms in the query, distances between query terms in the documents, attributes of tokens in the documents, and text that appears around a query term used in a document of the first set of documents;
• and
• storing the second set of relevancy scores in the memory;
• reading the first and second set of relevancy scores from the memory, and generating an ordered list of documents for further processing based on the first and second set of relevancy scores;
• automatically generating additional query terms from the documents in the ordered list of documents;
• formulating a new query using the additional query terms;
• processing the new query to retrieve a second set of document identifiers from the index and to generate a third set of relevancy scores based at least in part on the additional query terms;
• and
• using the third set of relevancy scores to select a set of top documents for presentation to the user.