Robust stream filtering based on reference document

A computer-implemented method for document stream filtering, comprising:

• receiving and storing one or more documents of a document stream in a search index;
• calculating global document frequencies for the one or more documents of the document stream;
• receiving one or more reference documents from a user terminal;
• generating a set of weighted, relevant terms from document frequency metrics of both the reference documents and the global documents;
• generating an input query by classifying the set of weighted, relevant terms into first and second categories based on individual term weights by:
• ordering the terms with decreasing term weight;
• initializing a clustering algorithm with individual term weights;
• determining whether a single cluster contains the top ranked term;
• assigning the second category to each term if the single cluster does not contain the top ranked term;
• determining whether the single cluster contains more or less than N terms when the single cluster contains the top ranked term;
• assigning the second category to each term, if the single cluster contains more than N terms;
• and
• assigning the first category to the first M terms, if the single cluster contains less than N terms, and assigning the second category to the remaining terms;
• retrieving one or more documents from the search index when the one or more documents contain a term of the first category;
• arranging the one or more retrieved documents in a decreasing order of relevance based on the presence of terms of the second category;
• and
• filtering the one or more retrieved documents based on a cut-off score and a last-seen timestamp variable, wherein the filtered documents have a numeric score greater than or equal to the cut-off score and with a creation timestamp newer than the last-seen timestamp variable.

A document stream filtering system, comprising:

• a receiving module operable to receive and store one or more documents of a document stream in a search index;
• a global document frequency module operable to calculate global document frequencies for the one or more documents of the document stream;
• a feature selection module operable to receive one or more reference documents from a user terminal and generate a set of relevant terms and corresponding weights using metrics of document frequencies of the one or more reference documents and the global document frequencies;
• a query term classification module operable to generate a query by classifying the set of relevant terms into first and second categories, wherein a clustering algorithm is used for identifying terms for each category based on the individual term weights by:
• ordering the terms with decreasing term weight;
• initializing a clustering algorithm with individual term weights;
• determining whether a single cluster contains the top ranked term;
• assigning the second category to each term if the single cluster does not contain the top ranked term;
• determining whether the single cluster contains more or less than N terms when the single cluster contains the top ranked term;
• assigning the second category to each term, if the single cluster contains more than N terms;
• and
• assigning the first category to the first M terms, if the single cluster contains less than N terms, and assigning the second category to the remaining terms;
• a document retrieval module operable to retrieve one or more documents from the search index when the one or more documents contain a term of the first category, arrange the one or more retrieved documents in a decreasing order of relevance based on the presence of terms of the second category;
• and a post-processing module operable to filter the one or more retrieved documents based on a cut-off score and a last-seen timestamp variable, wherein the filtered documents have a numeric score greater than or equal to the cut-off score and with a creation timestamp newer than the last-seen timestamp variable.