Systems and methods of de-duplicating similar news feed items

A method of de-duplicating similar news feed items, the method including:

• assembling a set of news feed items from a plurality of electronic sources;
• preprocessing the set to qualify some of the news feed items to return based on common company-name mentions and common token occurrences;
• pairwise determining a resemblance measure for the qualified news feed items based on results returned in response to supplying news feed item pairs as search criteria;
• constructing a graph of news feed item pairs with the resemblance measure above a threshold and representing the resemblance measure as edges between nodes representing the news feed item pairs, thereby forming connected node pairs;
• and
• determining similar news feed items by clustering the connected node pairs into strongly connected components.

A system of de-duplicating similar news feed items, the system including:

• a processor and a computer readable storage medium storing computer instructions configured to cause the processor to:
• assemble a set of news feed items from a plurality of electronic sources;
• preprocess the set to qualify some of the news feed items to return based on common company-name mentions and common token occurrences;
• pairwise determine a resemblance measure for the qualified news feed items based on sequence alignment between news feed item pairs;
• construct a graph of news feed item pairs with the resemblance measure above a threshold and representing the resemblance measure as edges between nodes representing the news feed item pairs, thereby forming connected node pairs;
• and
• determine similar news feed items by clustering the connected node pairs into strongly connected components.