Data locality based query optimization for scan operators

A system, comprising:

• a processor configured to:
• receive a query;
• in response to receiving the query, generate a query plan for the query, wherein the query plan includes a mapping between data blocks and segments to be used in connection with processing the query;
• identify a set of query processing segments to be used to process the query, comprising:
• determine for each of a plurality of data blocks to be scanned to process the query a corresponding storage location in which the data block is stored in a distributed storage system;
• and
• assign, for each of the plurality of data blocks to be scanned, a query processing segment that is selected based at least in part on an available bandwidth of the query processing segment and a location of the query processing segment relative to the corresponding storage location in which the data block is stored, comprising:
• determine if one or more segments on a host at which the data block is stored is available to process at least part of a query,
• in response to a determination that more than one segment on a host at which the data block is stored is available to process the at least the part of the query, assign the at least the part of the query to corresponding segment of the more than one segment that has a highest available bandwidth,
• determine for a next data block to be assigned whether any segment on a same host on which the data block is stored has an amount of available bandwidth that is greater than or equal to a size of the next data block, and
• assign, for the next data block, a corresponding query processing segment based at least in part on a determination of whether any segment on a same host on which the next data block is stored has the amount of available bandwidth that is greater than or equal to the size of the next data block;
• and
• communicate, to at least one processing segment from the set of query processing segments, a corresponding portion of the query plan and metadata including information indicating a location of data with respect to which processing is to be performed in connection with the query;
• and
• a memory or other storage device coupled to the processor and configured to store data indicating said corresponding storage location in which the data block is stored in the distributed storage system.

A computer program product embodied in a tangible, non-transitory computer-readable storage medium, comprising computer instructions for:

• receiving a query;
• in response to receiving the query, generating a query plan for the query, wherein the query plan includes a mapping between data blocks and segments to be used in connection with processing the query;
• identifying a set of query processing segments to be used to process the query, comprising:
• determining for each of a plurality of data blocks to be scanned to process the query a corresponding storage location in which the data block is stored in a distributed storage system;
• and
• assigning, for each of the plurality of data blocks to be scanned, a query processing segment that is selected based at least in part on an available bandwidth of the query processing segment and a location of the query processing segment relative to the corresponding storage location in which the data block is stored, comprising:
• determining if one or more segments on a host at which the data block is stored is available to process at least part of a query,
• in response to a determination that more than one segment on a host at which the data block is stored is available to process the at least the part of the query, assigning the at least the part of the query to corresponding segment of the more than one segment that has a highest available bandwidth,
• determining for a next data block to be assigned whether any segment on a same host on which the data block is stored has an amount of available bandwidth that is greater than or equal to a size of the next data block, and
• assigning, for the next data block, a corresponding query processing segment based at least in part on a determination of whether any segment on a same host on which the next data block is stored has the amount of available bandwidth that is greater than or equal to the size of the next data block;
• and
• communicating, to at least one processing segment from the set of query processing segments, a corresponding portion of the query plan and metadata including information indicating a location of data with respect to which processing is to be performed in connection with the query.