Incremental data retrieval based on structural metadata

An apparatus for reducing latency in processing a graph-based data retrieval query in a distributed database environment that comprises a plurality of data sources, the apparatus comprising at least one processor and at least one non-transitory memory including non-transitory program code, the at least one non-transitory memory and the non-transitory program code configured to, with the at least one processor, cause the apparatus to at least:

• identify a query definition model associated with the graph-based data retrieval query, wherein the query definition model comprises a graph-based structure and defines one or more requested hierarchical relationships between a plurality of requested data nodes for the graph-based data retrieval query;
• transmit a respective data retrieval request to each data source of the plurality of data sources to obtain a per-source portion of the plurality of requested data nodes that is associated with said each data source;
• determine a plurality of source-node pairs that are associated with the plurality of data sources and a requested data node of the plurality of requested data nodes;
• calculate a plurality of per-source node retrieval latency measures associated with the plurality of source-node pairs based on a historic data retrieval latency data table;
• compare the plurality of per-source node retrieval latency measures to identify a selected per-source node retrieval latency measure that indicates an improved latency in processing the graph-based data retrieval query;
• retrieve a data item corresponding to the requested data node from one of the plurality of data sources that corresponds to the selected per-source node retrieval latency measure;
• generate a query processing model for the graph-based data retrieval query that comprises the data item;
• calculate a model completeness measure for the query processing model that indicates a structural similarity measure between a modeling structure of the query definition model and a processing structure of the query processing model based on a hierarchical height measure defined between the data item in the query processing model and a root requested data node in the query definition model;
• determine whether the model completeness measure satisfies one or more completeness threshold criteria;
• in response to determining that the model completeness measure satisfies the one or more completeness threshold criteria, generate a partial query output for the graph-based data retrieval query based on the data item;
• and
• generate one or more user interface elements based at least in part on the partial query output.

A computer program product for reducing latency in processing a graph-based data retrieval query in a distributed database environment that comprises a plurality of data sources, the computer program product comprising at least one non-transitory computer-readable storage medium having non-transitory computer-readable program code portions stored therein, the non-transitory computer-readable program code portions configured to:

• identify a query definition model associated with the graph-based data retrieval query, wherein the query definition model comprises a graph-based structure and defines one or more requested hierarchical relationships between a plurality of requested data nodes for the graph-based data retrieval query;
• transmit a respective data retrieval request to each data source of the plurality of data sources to obtain a per-source portion of the plurality of requested data nodes that is associated with said each data source;
• determine a plurality of source-node pairs that are associated with the plurality of data sources and a requested data node of the plurality of requested data nodes;
• calculate a plurality of per-source node retrieval latency measures associated with the plurality of source-node pairs based on a historic data retrieval latency data table;
• compare the plurality of per-source node retrieval latency measures to identify a selected per-source node retrieval latency measure that indicates an improved latency in processing the graph-based data retrieval query;
• retrieve a data item corresponding to the requested data node from one of the plurality of data sources that corresponds to the selected per-source node retrieval latency measure;
• generate a query processing model for the graph-based data retrieval query that comprises the data item;
• calculate a model completeness measure for the query processing model that indicates a structural similarity measure between a modeling structure of the query definition model and a processing structure of the query processing model based on a hierarchical height measure defined between the data item in the query processing model and a root requested data node in the query definition model;
• determine whether the model completeness measure satisfies one or more completeness threshold criteria;
• in response to determining that the model completeness measure satisfies the one or more completeness threshold criteria, generate a partial query output for the graph-based data retrieval query based on the data item;
• and
• generate one or more user interface elements based at least in part on the partial query output.