Method and system for data mining

A non-transitory machine-readable medium having information recorded thereon for generating a stream of content items, wherein the information, when read by a machine, causes the machine to perform operations comprising:

• extracting, by an engine, one or more entities from each of published content items;
• determining a category of each of the published content items;
• appending each of the published content items with metadata including one or more of:
• a corresponding publisher, the corresponding one or more entities, the corresponding category, and a count of the corresponding one or more entities;
• storing the appended content items in a repository;
• evaluating the stored content items to determine trendiness scores for the exacted entities;
• and
• in response to a query for content items that are trending, generating a stream of content items based on the trendiness scores.

A system for generating a stream of content items, the system comprising:

• memory storing computer program instructions;
• and
• one or more processors that, in response to executing the computer program instructions, effectuate operations comprising:
• extracting, by an engine, one or more entities from each of published content items;
• determining a category of each of the published content items;
• appending each of the published content items with metadata including one or more of:
• a corresponding publisher, the corresponding one or more entities, the corresponding category, and a count of the corresponding one or more entities;
• storing the appended content items in a repository;
• evaluating the stored content items to determine trendiness scores for the exacted entities;
• and
• in response to a query for content items that are trending, generating a stream of content items based on the trendiness scores.