Load balanced access to distributed storage of data cubes

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

• receive node status from a plurality of node devices of a node device grid, wherein the node status comprises an indication of status of storage of each node device of the plurality of node devices;
• receive a request to perform an operation with a first data cube, wherein:
• the first data cube comprises a plurality of partitions;
• and
• at least one copy of the partitions of the first data cube are distributed among the plurality of node devices such that non-volatile storage of each node device stores a copy of at least one partition of the first data cube;
• compare quantities of the node devices of the plurality of node devices and of the partitions of the first data cube to determine whether the quantity of node devices of the plurality of node devices is less than the quantity of partitions of the first data cube;
• in response to a determination that the quantity of node devices of the plurality of node devices is less than the quantity of partitions of the first data cube:
• for each copy of each partition of the first data cube stored within the non-volatile storage of a node device of the plurality of node devices, assign a cost to load the copy of the partition from the non-volatile storage of the node device based at least on a size of the partition;
• add, to the quantity of node devices of the plurality of node devices, a first quantity of dummy node devices to cause a sum of the quantity of node devices of the plurality of node devices and the first quantity of dummy node devices to at least equal the quantity of partitions of the first data cube;
• for each dummy node device of the first quantity of dummy node devices, assign a cost to load each partition of the first data cube from a non-volatile storage of the dummy node device that is greater than any of the costs to load any partition of the first data cube stored in the non-volatile storage of any of the node devices of the plurality of node devices;
• use mixed integer linear programming (MILP), derive a first pass of assignment of a single copy of each partition of the first data cube to be loaded from the non-volatile storage of either a node device of the plurality of node devices or a dummy node device of the first quantity of dummy node devices;
• and
• transmit, to the plurality of node devices, task routines for execution by processors of the plurality of node devices to perform the requested operation, wherein each task routine comprises an indication of the assignment of a single copy of each partition of the first data cube to be loaded from the non-volatile storage of, and into the volatile storage of, a node device of the plurality of node devices based on at least the first pass of assignment.

A computer-implemented method comprising:

• receiving, at a coordinating device of a node device grid, node status from a plurality of node devices of the node device grid, wherein the node status comprises an indication of status of storage of each node device of the plurality of node devices;
• receiving, at the coordinating device, a request to perform an operation with a first data cube, wherein:
• the first data cube comprises a plurality of partitions;
• and
• at least one copy of the partitions of the first data cube are distributed among the plurality of node devices such that non-volatile storage of each node device stores a copy of at least one partition of the first data cube;
• comparing, at the coordinating device, quantities of the node devices of the plurality of node devices and of the partitions of the first data cube to determine whether the quantity of node devices of the plurality of node devices is less than the quantity of partitions of the first data cube;
• in response to a determination that the quantity of node devices of the plurality of node devices is less than the quantity of partitions of the first data cube, performing operations at the coordinating device comprising:
• for each copy of each partition of the first data cube stored within the non-volatile storage of a node device of the plurality of node devices, assigning a cost to load the copy of the partition from the non-volatile storage of the node device based at least on a size of the partition;
• adding, to the quantity of node devices of the plurality of node devices, a first quantity of dummy node devices to cause a sum of the quantity of node devices of the plurality of node devices and the first quantity of dummy node devices to at least equal the quantity of partitions of the first data cube;
• for each dummy node device of the first quantity of dummy node devices, assigning a cost to load each partition of the first data cube from a non-volatile storage of the dummy node device that is greater than any of the costs to load any partition of the first data cube stored in the non-volatile storage of any of the node devices of the plurality of node devices;
• using mixed integer linear programming (MILP), deriving a first pass of assignment of a single copy of each partition of the first data cube to be loaded from the non-volatile storage of either a node device of the plurality of node devices or a dummy node device of the first quantity of dummy node devices;
• and
• transmitting, to the plurality of node devices, task routines for execution by processors of the plurality of node devices to perform the requested operation, wherein each task routine comprises an indication of the assignment of a single copy of each partition of the first data cube to be loaded from the non-volatile storage of, and into the volatile storage of, a node device of the plurality of node devices based on at least the first pass of assignment.