System and method for discovering story trends in real time from user generated content

A system for discovering story trends, the system comprising:

• a plurality of client devices and a plurality of data sources coupled to a network;
• a web server operative to receive a fixed size data stream to and from the client devices and data sources;
• a word parser operative to receive a data stream, identify a first set of words within the data stream and determine which words in the first set of words are present in a word cache and updating the word cache based on the determination of which words in the first set of words are present in the word cache
• a subword parser operative to identify a second set of words within the data stream for each word present within the word cache and determine which words in the second set of words are present in a subword cache, updating the subword cache based on the determination of which words in the second set of words are present in the subword cache and identifying a third set of words based on the determination of which words in the second set of words are present in the subword cache, the subword cache including a pruned subset of the word cache based on one or more textual metrics;
• a hash calculator operative to determine at least one story trend associated with the third set of words and generate a story hash associated with the third set of words;
• a story trend cache operable to store the story hash and update the story trend cache according to the story hash;
• and
• a hamming calculator operable to retrieve one or more popular story topics according to the story trend cache.

Non-transitory machine readable media comprising program code that when executed by a programmable processor causes execution of a method for generating search results, the machine readable media including:

• program code for receiving a fixed size data stream;
• program code for identifying a first set of words within the data stream;
• program code for determining which words in the first set of words are present in a word cache and updating the word cache based on the determination of which words in the first set of words are present in the word cache;
• program code for identifying a second set of words within the data stream for each word present within the word cache;
• program code for determining which words in the second set of words are present in a subword cache, updating the subword cache based on the determination of which words in the second set of words are present in the subword cache and identifying a third set of words based on the determination of which words in the second set of words are present in the subword cache, the subword cache including a pruned subset of the word cache based on one or more textual metrics;
• program code for determining at least one story trend associated with the third set of words and generating a story hash associated with the third set of words;
• program code for storing the story hash in a story trend cache and updating the story trend cache according to the story hash;
• and
• program code for retrieving one or more popular story topics according to the story trend cache.