Data storage system with multi-tier control plane

A data storage system, comprising:

• a data storage unit assigned to service read requests and write requests directed to a volume partition stored in the data storage unit, the data storage unit comprising:
• a plurality of head nodes comprising:
• a designated-primary head node configured to service the volume partition;
• and
• one or more data storage sleds for storing data flushed from the head nodes;
• wherein the designated-primary head node is configured to:
• receive a recommendation that indicates availability of a sandbox of specified storage, allocated to the volume and in the data storage sleds, for selection, by the designated-primary head node, for data storage during a data flush from the designated-primary head node to the data storage sleds;
• select, from the sandbox of specified storage in the data storage sleds that is allocated to the volume, one or more portions of storage to store data during the data flush from the designated-primary head node to the data storage sleds;
• and
• flush data to the selected storage.

A data storage system, comprising:

• a head node of a data storage unit assigned to service read requests and write requests directed to a volume partition stored in a data storage unit, the data storage unit comprising:
• respective designated-primary head nodes configured to service respective volume partitions stored in the data storage unit;
• respective designated-secondary head nodes configured to replicate data of the corresponding designated-primary head nodes;
• and data storage sleds for storing data flushed from the designated-primary head nodes,
• wherein the head node is configured to implement a local control plane for the data storage unit, configured to:
• generate and transmit a recommendation of a sandbox of specified storage, associated with a respective volume partition, in the data storage sleds, to a respective designated-primary head node for the respective volume partition;
• wherein the recommendation indicates availability of the sandbox of specified storage in the data storage sleds for selection, by the respective designated-primary head node, for data storage during a data flush from the respective designated-primary head node to the data storage sleds.