Systems and methods for providing performance metadata in interest-driven business intelligence systems

An interest-driven business intelligence server system, comprising:

• a processor;
• a memory connected to the processor and configured to store an interest-driven business intelligence application;
• raw data storage configured to store raw data, wherein a portion of the raw data comprises unstructured data;
• metadata storage configured to store data description metadata describing the raw data;
• and
• report storage configured to store previously generated reports comprising previous reporting data and previous reporting data requirements;
• wherein the interest-driven business intelligence application configures the processor to:
• receive a report specification, wherein the report specification comprises at least one reporting data requirement;
• generate a sample of source data satisfying the at least one reporting data requirement from a sample of the raw data;
• determine estimates of performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based at least in part on the report specification and the data description metadata describing the raw data, wherein the estimates of performance metadata information comprise an estimated data size of reporting data, wherein the determining of the performance metadata information satisfies the at least one reporting data requirement, and wherein the determining of the estimates of performance metadata information comprises to:
• apply required filters to the sample of the raw data;
• apply required aggregations to the sample of the raw data;
• determine performance metadata for the sample of source data, wherein the performance metadata includes performance metadata for the required aggregations and required filters and the required aggregations;
• and
• determine the estimates of performance metadata for the sample of reporting data satisfying the at least one reporting data requirement based on the determined performance metadata of the sample of source data;
• wherein the performance metadata includes performance metadata for the required aggregations and required filters and the required aggregations
• provide the estimates of the performance metadata information for the interest-driven data pipeline to an interest-driven user visualization system, wherein the interest-driven user visualization system displays the estimated data size of the reporting data;
• generate the reporting data utilizing the interest-driven data pipeline;
• and
• store the reporting data and the at least one reporting data requirement in the report storage.