Apparatus, method, and computer program product for distributed semantic search

A computer-implemented method comprising:

• partitioning a main text data set representing a master corpus into a plurality of node text data sets each representing a portion of the master corpus based at least in part on a predefined partitioning schema, wherein each node text data set of the plurality of node text data sets corresponds uniquely to a semantic search node of a plurality of semantic search nodes;
• for each semantic search node of the plurality of semantic search nodes:
• selecting, from a plurality of index structures associated with a semantic search process, an optimal index structure based at least in part on hyperparameters associated with the plurality of index structures;
• and
• generating a node index based at least in part on the node text data set corresponding to the semantic search node according to selected optimal index structure;
• storing the plurality of node indexes in a node index store as a file in a columnar file format comprising a table in which each of the node indexes is stored as a corresponding column of binary values;
• for each semantic search node of the plurality of semantic search nodes, loading the corresponding node index from the node index store into a persistent node index cache of the semantic search node, wherein the node index persists independently with respect to distinct executions of a corresponding instance of a semantic search;
• receiving a query vector representing input for the semantic search over the master corpus;
• at each semantic search node of the plurality of semantic search nodes, generating a node results set representing output of the semantic search over the portion of the master corpus represented by the node text data set corresponding to the semantic search node by executing an instance of a semantic search process based at least in part on the query vector and on the node index corresponding to the semantic search node;
• and
• generating a master results set representing output of the semantic search over the master corpus by combining a plurality of node results sets comprising the node results set generated at each semantic search node of the plurality of semantic search nodes.

A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising an executable portion configured to:

• partition a main text data set representing a master corpus into a plurality of node text data sets each representing a portion of the master corpus based at least in part on a predefined partitioning schema, wherein each node text data set of the plurality of node text data sets corresponds uniquely to a semantic search node of a plurality of semantic search nodes;
• for each semantic search node of the plurality of semantic search nodes:
• select, from a plurality of index structures associated with a semantic search process, an optimal index structure based at least in part on hyperparameters associated with the plurality of index structures;
• and
• generate a node index based at least in part on the node text data set corresponding to the semantic search node according to the selected optimal index structure;
• store the plurality of node indexes in a node index store as a file in a columnar file format comprising a table in which each of the node indexes is stored as a corresponding column of binary values;
• for each semantic search node of the plurality of semantic search nodes, load the corresponding node index from the node index store into a persistent node index cache of the semantic search node, wherein the node index persists independently with respect to distinct executions of a corresponding instance of a semantic search;
• receive a query vector representing input for the semantic search over the master corpus;
• at each semantic search node of the plurality of semantic search nodes, generate a node results set representing output of a semantic search over the portion of the master corpus represented by the node text data set corresponding to the semantic search node by executing an instance of a semantic search process based at least in part on the query vector and on the node index corresponding to the semantic search node;
• and
• generate a master results set representing output of a semantic search over the master corpus by combining a plurality of node results sets comprising the node results set generated at each semantic search node of the plurality of semantic search nodes.