Methods and systems for identifying a level of similarity between a filtering criterion and a data item within a set of streamed documents

A computer-implemented method for identifying a level of similarity between a user provided data item and a data item within a set of data documents, the method comprising:

• clustering, by a reference map generator executing on a first computing device, in a two-dimensional metric space, a set of data documents selected according to at least one criterion, generating a semantic map;
• associating, by the semantic map, a coordinate pair with each data document of the set of data documents;
• generating, by a parser executing on the first computing device, an enumeration of terms occurring in the set of data documents;
• determining, by a representation generator executing on the first computing device, for each term in the enumeration, occurrence information including:
• (i) a number of data documents in which the term occurs, (ii) a number of occurrences of the term in each data document, and (iii) the coordinate pair associated with each data document in which the term occurs;
• generating, by the representation generator, for each term in the enumeration, a sparse distributed representation (SDR) using the occurrence information;
• storing, in an SDR database, each of the generated SDRs;
• receiving, by a filtering module executing on a second computing device and in communication with the first computing device, from a third computing device, a filtering criterion;
• generating, by the representation generator, for the filtering criterion, at least one SDR, wherein generating further comprises:
• determining, by the representation generator, that the filtering, criterion is not an SDR stored in the SDR database:
• and
• generating. by the representation generator, the at least one SDR for the filtering criterion, based upon the determining that the filtering criterion is not an SDR;
• receiving, by the filtering module, a first plurality of streamed documents from a third-party data source specified by a user of the third computing device;
• generating, by the representation generator, for a first document in the first plurality of streamed documents, a compound SDR for the first document in the first plurality of streamed documents, before receiving, by the filtering module, a second document in the first plurality of streamed documents;
• determining, by a similarity engine executing on the second computing device, a distance between the filtering criterion SDR and the generated compound SDR for the first document in the first plurality of streamed documents;
• and
• acting, by the filtering module, on the first of the first plurality of streamed documents, based upon the determined distance, wherein acting further comprises:
• adding the first document in the first plurality of streamed. documents to a sub-stream of streamed documents stored in a database accessible by a client agent executing on the third computing device;
• and
• responding to a polling request from the client agent by transmitting the sub-stream to the client agent.