Dynamic scheduling of distributed storage management tasks using predicted system characteristics

A non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a distributed system, causes the distributed system to perform a set of acts, the set of acts comprising:

• maintaining multiple storage devices as a storage pool in the distributed system, a first computing node and a second computing node each comprising a respective virtual machine that accesses the storage pool, wherein the first computing node comprises a first processor, a first memory, and first storage device of the multiple storage devices, the second computing node comprises a second processor, a second memory, and second storage device of the multiple storage devices, the first and second computing nodes being interconnected over a network;
• generating a prediction model with respect to execution of a first task that executes on the first computing node, wherein the execution of the first task causes consumption of a first resource at the first computing node and a second resource at the second computing node, the generating comprising:
• receiving a data record that characterizes multiple tasks executed in the distributed system, wherein the data record characterizes at least one task that was executed at the first computing node, and the at least one task when executed at the first computing node results in an effect on a first metric for the first computing node and a second metric for the second computing node;
• generating the prediction model based on at least the data record that characterizes the at least one task that was executed at the first computing node that results in the effect on the first metric for the first computing node and the second metric for the second computing node;
• identifying a second task to be executed based on the prediction model generated with respect to the execution of the first task by:
• generating a set of predicted metrics for at least one forecasted task of a set of forecasted tasks using at least the prediction model;
• and
• selecting the second task separate from the set of forecasted tasks to be executed on the distributed system.

A system, comprising:

• a non-transitory storage medium having stored thereon a sequence of instructions;
• and
• a processor that executes the sequence of instructions, execution of the sequence of instructions causing the processor to perform a set of acts, the set of acts comprising:
• maintaining multiple storage devices as a storage pool in a distributed system, a first computing node and a second computing node each comprising a respective virtual machine that accesses the storage pool, wherein the first computing node comprises a first processor, a first memory, and first storage device of the multiple storage devices, the second computing node comprises a second processor, a second memory, and second storage device of the multiple storage devices, the first and second computing nodes being interconnected over a network;
• generating a prediction model with respect to execution of a first task that executes on the first computing node, wherein the execution of the first task causes consumption of a first resource at the first computing node and a second resource at the second computing node, the generating comprising:
• receiving a data record that characterizes multiple tasks executed in the distributed system, wherein the data record characterizes at least one task that was executed at the first computing node, and the at least one task when executed at the first computing node results in an effect on a first metric for the first computing node and a second metric for the second computing node;
• generating the prediction model based on at least the data record that characterizes the at least one task that was executed at the first computing node that results in the effect on the first metric for the first computing node and the second metric for the second computing node;
• identifying a second task to be executed based on the prediction model generated with respect to the execution of the first task by:
• generating a set of predicted metrics for at least one forecasted task of a set of forecasted tasks using at least the prediction model;
• and
• selecting the second task separate from the set of forecasted tasks to be executed on the distributed system.