Efficient large-scale filtering and/or sorting for querying of column based data encoded structures

A computer-implemented method for processing an external query request received from an application, the processing of the external query request being carried out in a manner that reduces the amount of data that must be processed, thereby more efficiently returning results for the external query request, the computer-implemented method comprising:

• at one or more processors of a computing system, receiving an external query request from an application, wherein the external query request requests to apply operations over a data set in a data store, including a windowing operation that defines a window of a subset of sorted and filtered data that is to be returned after both a filter operation and a sort operation are applied to the data set;
• at one or more processors of the computing system, splitting the external query request into a first internal query request and a second internal query request, including performing the following:
• generating the first internal query request, which is configured to generate query-specific distribution statistics that are usable in determining what sorted and filtered data would be returned by the windowing operation, the distribution statistics including statistics that are specific to said external query request, the distribution statistics containing information about a distribution of rows within the data store that match the one or more operations;
• executing the first internal query request to generate the distribution statistics;
• and
• using the distribution statistics to generate the second internal query request, the second internal query request being configured to:
• execute said filter and sort operations of the external query request while limiting the filter and sort operations to the subset of data defined by the windowing operation of the external query request, such that sorting comprises sorting only the number of rows associated with a size of the requested window over the data set, and such that the second internal sub-request selects only the rows that match the window based on the distribution statistics;
• and
• return at least one result set for the external query request;
• the computing system executing the second internal query request to generate the at least one result set;
• and
• the computing system transmitting the at least one result set to the application.

One or more computer storage media having stored thereon computer executable instructions that, when executed by one or more processing units of a computer system, implement a method for more efficiently processing a query request, the method comprising:

• receiving an external query request from an application, the external query request requesting to apply operations over a data set, including both a filter operation and a sort operation, and to retrieve a subset of data matching the operations, as defined by a windowing operation;
• generating at least one first internal query request and at least one second internal query request from the external query request, including:
• generating the first internal query request, which is configured to compute query-specific distribution statistics, specific to the external query request, about a distribution of rows and columns within the data set that are implicated by the operations, including rows matching the filter operation and columns matching the sort operation;
• executing the first internal query request to generate the query-specific distribution statistics;
• and
• generating the second internal query request based on the previously computed query-specific distribution statistics, the second internal query request being configured to:
• execute said filter and sort operations of the external query request while limiting the filter and sort operations to the subset of data defined by the windowing operation of the external query request, such that sorting comprises sorting only the number of rows associated with a size of a requested window over the data set, and such that the second internal sub-request selects only the rows that match the window based on the distribution statistics;
• and
• return at least one result set for the external query request;
• executing the second internal query request to generate the at least one result set;
• transmitting the at least one result set to the application.