Skew detection and handling in a parallel processing relational database system
AssigneeTeradata US, Inc.
InventorsBhashyam Ramesh, Suresh Kumar Jami
Life of this patent
Estimating the term…
Abstract
A method for detecting and handling skew and spillover in in-memory hash join operations. To improve the detection of skew and spillover in parallel processing systems, a Poisson distribution of unique hash values to Units of Parallelism (UoPs) is employed to determine the number of rows per UoP and in turn, the potential of spillover at a UoP. Hash join plan options can be selected or adjusted to reduce the likelihood of spillover.
Claims10
A method for reducing spillover during a hash join in a parallel processing relational database system, said system comprising a plurality of Units of Parallelism (UoPs), said method comprising the steps of:
- evaluating, by a processor, multiple plan options for said hash join, wherein evaluating comprises for each one of said multiple plan options:
- determining a highest number of rows to be distributed to a UoP using a Poisson distribution of the number of unique hash values to the number of UoPs;
- and
- determining a probability of spillover at said UoP from said highest number of rows;
- selecting, by said processor, from said multiple plan options, a plan option having the lowest probability for spillover;
- and
- executing, by said processor, said hash join in accordance with said selected plan option.
A method for reducing cost of a hash join in a parallel processing relational database system, said system comprising a plurality of Units of Parallelism (UoPs), said method comprising the steps of:
- evaluating, by a processor, multiple plan options for said hash join, wherein evaluating comprises for each one of said multiple plan options:
- determining a highest number of rows to be distributed to a UoP using a Poisson distribution of the number of unique hash values to the number of UoPs;
- and
- determining a cost associated with said highest number of rows;
- selecting, by said processor, from said multiple plan options, a plan option having the lowest cost;
- and
- executing, by said processor, said hash join in accordance with said selected plan option.
A method for estimating cardinality skew for an output relation and reducing spillover during a hash join in a parallel processing relational database system, said system comprising a plurality of Units of Parallelism (UoPs), said method comprising the steps of:
- determining, by a processor, if one or more input relations are skewed;
- if one or more input relations are skewed, determining, by said processor, if statistics are available for the skewed input relations;
- and
- if statistics are not available for at least one of the skewed input relations, identifying, by said processor, said output relation as skewed.
Dependencies are inferred from the claim text.