# US 11,442,660 B2 · Dynamic scheduling of distributed storage management tasks using predicted system characteristics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,660 B2 - Document ID: US11442660B2 - Status: Granted - Kind: B2 - Application number: 16/237,450 - Filed: 2018-12-31 - Published: 2019-05-09 - Granted: 2022-09-13 - Earliest claimed: 2016-05-20 - Assignees: Nutanix, Inc. - Applicants: Nutanix, Inc. - Inventors: Karan Gupta; Varun Kumar Arora; Himanshu Shukla; Bharat Kumar Beedu; Abhinay Nagpal - Web page: https://patentgrep.com/patent/US11442660B2 ## CPC - G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling} - G06F 3/0604: {Improving or facilitating administration, e.g. storage management} - G06F 3/0605: {by facilitating the interaction with a user or administrator} - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 3/0629: {Configuration or reconfiguration of storage systems} - G06F 3/0653: {Monitoring storage devices or systems} - G06N 20/20: Ensemble learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for scheduling storage management tasks over predicted user tasks in a distributed storage system. A method commences upon receiving a set of historical stimulus records that characterize management tasks that are run in the storage system. A corresponding set of historical response records comprising system metrics associated with execution of the system tasks is also received. A learning model is formed from the stimulus records and the response records and formatted to be used as a predictor. A set of forecasted user tasks is input as new stimulus records to the predictor to determine a set of forecasted system metrics that would result from running the forecasted user tasks. Management tasks are selected so as not to impact the forecasted user tasks. Management tasks can be selected based on non-contentions resource usage between historical management task resource usage and predictions of resource usage by the user tasks. ## Claims ### Claim 1 A method, comprising: maintaining multiple storage devices as a storage pool in a distributed system, a first computing node and a second computing node each comprising a respective virtual machine that accesses the storage pool, wherein the first computing node comprises a first processor, a first memory, and first storage device of the multiple storage devices, the second computing node comprises a second processor, a second memory, and second storage device of the multiple storage devices, the first and second computing nodes being interconnected over a network; generating a prediction model with respect to execution of a first task that executes on the first computing node, wherein the execution of the first task causes consumption of a first resource at the first computing node and a second resource at the second computing node, the generating comprising: receiving a data record that characterizes multiple tasks executed in the distributed system, wherein the data record characterizes at least one task that was executed at the first computing node, and the at least one task when executed at the first computing node results in an effect on a first metric for the first computing node and a second metric for the second computing node; generating the prediction model based on at least the data record that characterizes the at least one task that was executed at the first computing node that results in the effect on the first metric for the first computing node and the second metric for the second computing node; identifying a second task to be executed based on the prediction model generated with respect to the execution of the first task by: generating a set of predicted metrics for at least one forecasted task of a set of forecasted tasks using at least the prediction model; and selecting the second task separate from the set of forecasted tasks to be executed on the distributed system. ### Claim 2 (depends on 1) The method of claim 1, wherein the second task corresponds to a management task type and is scheduled for the execution at the first computing node and consumes computing resources at the second computing node, and the at least one forecasted task corresponds to a user task type. ### Claim 3 (depends on 1) The method of claim 1, further comprising selecting the second task to be scheduled for execution in response to a prediction that forecasted resource consumption by the at least one forecasted task on the second computing node is to exceed a resource consumption limit at a future time point or in a future time period. ### Claim 4 (depends on 1) The method of claim 1, wherein the prediction model characterizes a correlation between the at least one forecasted task and system metrics or between the second task and the system metrics. ### Claim 5 (depends on 1) The method of claim 1, wherein the multiple storage devices that form the storage pool comprise a first local storage device on the first computing node and a second local storage device on the second computing node, the first local storage device accessible by the second computing node through the storage pool, and the second local storage device accessible by the first computing node through the storage pool. ### Claim 6 (depends on 1) The method of claim 1, wherein the second task to be scheduled for execution on the distributed system is managed by a controller virtual machine on the first computing node or the second computing node. ### Claim 7 (depends on 1) The method of claim 1, wherein the second task is selected based at least in part a correlation that has been determined between the second task and at least one of the first metric or the second metric by using at least the data record, and the correlation comprises at least one of a positive correlation or a negative correlation. ### Claim 8 A non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a distributed system, causes the distributed system to perform a set of acts, the set of acts comprising: maintaining multiple storage devices as a storage pool in the distributed system, a first computing node and a second computing node each comprising a respective virtual machine that accesses the storage pool, wherein the first computing node comprises a first processor, a first memory, and first storage device of the multiple storage devices, the second computing node comprises a second processor, a second memory, and second storage device of the multiple storage devices, the first and second computing nodes being interconnected over a network; generating a prediction model with respect to execution of a first task that executes on the first computing node, wherein the execution of the first task causes consumption of a first resource at the first computing node and a second resource at the second computing node, the generating comprising: receiving a data record that characterizes multiple tasks executed in the distributed system, wherein the data record characterizes at least one task that was executed at the first computing node, and the at least one task when executed at the first computing node results in an effect on a first metric for the first computing node and a second metric for the second computing node; generating the prediction model based on at least the data record that characterizes the at least one task that was executed at the first computing node that results in the effect on the first metric for the first computing node and the second metric for the second computing node; identifying a second task to be executed based on the prediction model generated with respect to the execution of the first task by: generating a set of predicted metrics for at least one forecasted task of a set of forecasted tasks using at least the prediction model; and selecting the second task separate from the set of forecasted tasks to be executed on the distributed system. ### Claim 9 (depends on 8) The computer readable medium of claim 8, wherein the second task corresponds to a management task type and is scheduled for the execution at the first computing node and consumes computing resources at the second computing node, and the at least one forecasted task corresponds to a user task type. ### Claim 10 (depends on 9) The computer readable medium of claim 9, wherein the set of acts further comprise selecting the second task to be scheduled for execution in response to a prediction that forecasted resource consumption by the at least one forecasted task on the second computing node is to exceed a resource consumption limit at a future time point or in a future time period. ### Claim 11 (depends on 8) The computer readable medium of claim 8, wherein the prediction model characterizes a correlation between the at least one forecasted task and system metrics or between the second task and the system metrics. ### Claim 12 (depends on 8) The computer readable medium of claim 8, wherein the multiple storage devices that form the storage pool comprise a first local storage device on the first computing node and a second local storage device on the second computing node, the first local storage device accessible by the second computing node through the storage pool, and the second local storage device accessible by the first computing node through the storage pool. ### Claim 13 (depends on 8) The computer readable medium of claim 8, wherein the second task to be scheduled for execution on the distributed system is managed by a controller virtual machine on the first computing node or the second computing node. ### Claim 14 (depends on 8) The computer readable medium of claim 8, wherein the second task is selected based at least in part a correlation that has been determined between the second task and at least one of the first metric or the second metric by using at least the data record, and the correlation comprises at least one of a positive correlation or a negative correlation. ### Claim 15 A system, comprising: a non-transitory storage medium having stored thereon a sequence of instructions; and a processor that executes the sequence of instructions, execution of the sequence of instructions causing the processor to perform a set of acts, the set of acts comprising: maintaining multiple storage devices as a storage pool in a distributed system, a first computing node and a second computing node each comprising a respective virtual machine that accesses the storage pool, wherein the first computing node comprises a first processor, a first memory, and first storage device of the multiple storage devices, the second computing node comprises a second processor, a second memory, and second storage device of the multiple storage devices, the first and second computing nodes being interconnected over a network; generating a prediction model with respect to execution of a first task that executes on the first computing node, wherein the execution of the first task causes consumption of a first resource at the first computing node and a second resource at the second computing node, the generating comprising: receiving a data record that characterizes multiple tasks executed in the distributed system, wherein the data record characterizes at least one task that was executed at the first computing node, and the at least one task when executed at the first computing node results in an effect on a first metric for the first computing node and a second metric for the second computing node; generating the prediction model based on at least the data record that characterizes the at least one task that was executed at the first computing node that results in the effect on the first metric for the first computing node and the second metric for the second computing node; identifying a second task to be executed based on the prediction model generated with respect to the execution of the first task by: generating a set of predicted metrics for at least one forecasted task of a set of forecasted tasks using at least the prediction model; and selecting the second task separate from the set of forecasted tasks to be executed on the distributed system. ### Claim 16 (depends on 15) The system of claim 15, wherein the second task corresponds to a management task type and is scheduled for the execution at the first computing node and consumes computing resources at the second computing node, and the at least one forecasted task corresponds to a user task type. ### Claim 17 (depends on 15) The system of claim 15, wherein the prediction model characterizes a correlation between the at least one forecasted task and system metrics or between the second task and the system metrics. ### Claim 18 (depends on 15) The system of claim 15, wherein the multiple storage devices that form the storage pool comprise a first local storage device on the first computing node and a second local storage device on the second computing node, the first local storage device accessible by the second computing node through the storage pool, and the second local storage device accessible by the first computing node through the storage pool. ### Claim 19 (depends on 15) The system of claim 15, wherein the second task to be scheduled for execution on the distributed system is managed by a controller virtual machine on the first computing node or the second computing node. ### Claim 20 (depends on 19) The system of claim 19, wherein the second task is selected based at least in part a correlation that has been determined between the second task and at least one of the first metric or the second metric by using at least the data record, and the correlation comprises at least one of a positive correlation or a negative correlation. ### Claim 21 (depends on 19) The system of claim 19, wherein the set of acts further comprise selecting the second task to be scheduled for execution in response to a prediction that forecasted resource consumption by the at least one forecasted task on the second computing node is to exceed a resource consumption limit at a future time point or in a future time period. ### Claim 22 (depends on 1) The method of claim 1, wherein the virtual machine comprises a controller virtual machine. ### Claim 23 (depends on 1) The method of claim 1, wherein the virtual machine comprises a user virtual machine. ### Claim 24 (depends on 8) The computer readable medium of claim 8, wherein the virtual machine comprises a controller virtual machine. ### Claim 25 (depends on 8) The computer readable medium of claim 8, wherein the virtual machine comprises a user virtual machine. ### Claim 26 (depends on 19) The system of claim 19, wherein the virtual machine comprises a controller virtual machine. ### Claim 27 (depends on 19) The system of claim 19, wherein the virtual machine comprises a user virtual machine.