Finding Dimensional Correlation Using Hyperloglog

A system comprising:

• memory storing instructions;
• and
• one or more processors coupled to the memory and configured to execute the stored instructions to:
• receive a first search term;
• access a first hyperloglog sketch of the first search term and a second hyperloglog sketch of a second search term;
• and
• determine a degree of overlap between the first search term and the second search term based on the first hyperloglog sketch of the first search term and a second hyperloglog sketch of a second search term,
• wherein the one or more processors are configured to execute the stored instructions to determine the degree of overlap between the first search term and the second search term by:
• merging the first and second hyperloglog sketches;
• and
• determining a third count of distinct elements indexed under the first and second search terms based on the merged first and second hyperloglog sketches, wherein the degree of overlap between the first search term and the second search term is based further on the third count of distinct elements.

A method comprising:

• receiving, by one or more processors, information indicative of data elements included in a plurality of tables stored remotely from the one or more processors;
• creating, by the one or more processors, a search index from the received information, wherein the search index includes a plurality of terms derived from the received information;
• determining, by the one or more processors, for each term, a count of distinct elements including the term according to a hyperloglog algorithm;
• generating, by the one or more processors, for each term, a hyperloglog sketch of the term according to the determined distinct count of elements, wherein the hyperloglog sketch is arranged to indicate a degree of overlap between the term and another term of the search index based on a merged count of distinct elements determined from a merge of the hyperloglog sketches of the term and the other term;
• and
• storing, by the one or more processors, for each term, the determined count of distinct elements including the term and the hyperloglog sketch of the term in the search index.