Two level QoS scheduling for latency and queue depth control

A tangible, non-transitory, computer-readable media having instructions thereupon which, when executed by a processor, cause the processor to perform a method for two level quality of service scheduling for latency and queue depth control in a storage system having a plurality of storage subsystems in which each storage subsystem has an associated queue and a level one scheduler provides a first level of scheduling input/output interactions with storage clients based on overall storage system capacity and a level two scheduler manages input/output flows with individual storage subsystems, the method comprising:

• determining by the level two scheduler, for each storage subsystem of a plurality of storage subsystems of a storage system how many input/output operations per unit time the storage subsystem can sustain, as a queue depth of the storage subsystem, wherein the queue depth is associated with a queue coupled to the storage subsystem, there being a plurality of such queues corresponding to the plurality of storage subsystems;
• determining, by the level one scheduler, based on the queue depths of the plurality of storage subsystems, total input/output operations per unit time that the storage system can sustain, as system capacity of the storage system;
• accepting or refusing, by the level two scheduler, to place input/output requests onto an input/output stack based on credits of each of the input/output requests in comparison to system capacity determined on an ongoing basis;
• moving, by the level two scheduler, the input/output requests from the input/output stack to the plurality of queues corresponding to the plurality of storage subsystems so as to maintain the queue depth of each such queue;
• controlling latency of input/output request servicing on the ongoing basis, by the accepting or refusing to place the input/output requests onto the input/output stack;
• and
• controlling the queue depth of each of the plurality of queues, by the determining the queue depth and each decision about when, and to which queue, to perform the moving of each of the input/output requests from the input/output stack to one of the plurality of queues;
• wherein the level one schedler regulates congestion of the input/output stack beyond the capacity of the storage system and the level two scheduler regulates oscillations in throughput.

A storage system scheduler, comprising:

• an input/output stack;
• a plurality of queues, each configured to couple to one of a plurality of storage subsystems of a storage system;
• one or more processors, configured to implement a level one scheduler and a level two scheduler in which the level one scheduler provides a first level of scheduling input/output interactions with storage clients based on overall storage system capacity and the level two scheduler manages input/output flows with individual storage subsystems;
• the level two scheduler configured to determine, on a continuous or interval basis, queue depth for each queue of the plurality of queues, wherein the queue depth is input/output operations per unit time the one of the plurality of storage subsystems to which the queue is coupled can sustain;
• the level one scheduler configured to determine, on the continuous or interval basis, system capacity of the storage system, wherein the system capacity is the total input/output operations per unit time that the storage system can sustain, based on the queue depth of each of the plurality of queues;
• the level one scheduler configured to compare credits of input/output requests to the system capacity as determined on the continuous or interval basis and accept or reject input/output requests onto the input/output stack based on such comparison;
• the level two scheduler configured to move the input/output requests from the input/output stack onto the plurality of queues based on maintaining the queue depth of each of the plurality of queues;
• the level two scheduler configured to control the queue depth of each of the plurality of queues on a continuous or interval basis;
• wherein the level one scheduler regulates congestion of the input/output stack beyond the capacity of the storage system and the level two scheduler regulates oscillations in throughput.