Unsupervised information-based hierarchical clustering of big data

A system, comprising:

• storage configured to store a data set comprising a plurality of data points;
• a processor;
• and
• a hierarchical clustering engine, implemented as machine-readable instructions stored in a memory in electronic communication with the processor, that, when executed by the processor, control the system to:
• stochastically partition a data set to create a plurality of pseudo-partitions of the data set, each of the plurality of pseudo-partitions comprising a plurality of subsets whose union is a subset of the data set,
• form each of a plurality of candidate sets based on one or more subsets selected from the plurality of subsets of each pseudo-partition of two or more pseudo-partitions of the plurality of pseudo-partitions,
• create each of a plurality of intersection sets based on two or more candidate sets selected from the plurality of candidate sets,
• identify, with each of the plurality of intersection sets, one or more co-occurrences of a plurality of co-occurrences of the plurality of data points,
• construct a graph having a plurality of vertices connected by a plurality of edges, each vertex of the plurality of vertices representing a respective one of the plurality of data points, each edge of the plurality of edges connecting two vertices of the plurality of vertices, said each edge representing a number of the co-occurrences for the two data points represented by the two vertices,
• identify a community in the graph, the community comprising member vertices of the plurality of vertices, and
• output a cluster of the data set, the cluster comprising member data points, of the plurality of data points, that are represented by the member vertices of the community.