On-demand storage provisioning using distributed and virtual namespace management

A storage system comprising:

• a host computer;
• and
• a plurality of storage nodes connected to the host computer over a network,
• wherein the plurality of storage nodes includes a storage node comprising one or more local storage devices and a storage node controller,
• wherein the storage node controller comprises:
• a host interface configured to connect to a client application running on the host computer;
• a storage manager configured to manage one or more virtual namespaces;
• and
• a storage device controller configured to access the one or more local storage devices of the storage node and manage respective namespace associated with the one or more storage devices,
• wherein the storage manager is further configured to provide mapping the one or more virtual namespaces to the respective namespace associated with the one or more storage devices,
• wherein the storage manager is further configured to determine that a current storage space associated with a virtual namespace used by the client application is insufficient to store data in response to a request by the client application and expand the storage space of the storage node associated with the virtual namespace on the one or more local storage devices of the storage node based on consumption of the current storage space by the client application, and
• wherein the storage manager is further configured to determine that the storage node has an insufficient storage space, communicate with a peer storage manager of a second storage node over a network and expand the storage space associated with the virtual namespace used by the client application on a local storage device of the second storage node in addition to the current storage space allocated in the storage node.

A method comprising:

• allocating a storage space associated with a virtual namespace in a first storage device of a first storage node;
• receiving a write request to store data on the storage space from a client application running on a host computer;
• determining that the storage space allocated in the first storage device has an insufficient storage space to store the data associated with the write request;
• in response to the write request, expanding the storage space associated with the virtual namespace on the first storage device based on current consumption of the storage space in the first storage device that is used by the client application by allocating an additional storage space in the first storage node to store the data;
• writing the data to the additional storage space in the first storage node;
• identifying a second storage node that has an additional storage space to store the data by determining that the first storage device has insufficient storage space;
• negotiating with the second storage node to allocate the additional storage space;
• expanding the storage space associated with the virtual namespace by allocating the additional storage space in the second storage node;
• writing the data to the additional storage space in the second storage node by peer-to-peer communication between the first storage node and the second storage node via the network;
• and
• updating a global mapping table to update mapping information of the virtual namespace.