Cloud capacity scaling with high readable capacity in metadata space constrained deduplication systems

A system for scaling capacity in a deduplicated storage appliance having limited metadata space, the system comprising:

• a processor;
• and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
• maintaining a plurality of cloud units in a cloud storage, each cloud unit comprising a plurality of containers comprising data containers storing segments of files, segment tree containers storing upper-level segments of segment trees representing the files, and cloud containers storing headers from the data and segment tree containers, wherein a header for a data container comprises fingerprints identifying the segments of the files, and a header for a segment tree container comprises fingerprints identifying the upper-level segments;
• storing, at local storage of the deduplicated storage appliance, a first level of metadata for each cloud unit in a read-write state, a second level of metadata for each cloud unit in a read-only state, and a third level of metadata for each cloud unit in an offline state, the third level of metadata requiring less storage space than the first and second levels of metadata, and the second level of metadata requiring less storage space than the first level of metadata;
• managing the limited metadata space on the deduplicated storage appliance by maintaining at least a subset of the plurality of cloud units in the read-only state;
• and
• for each cloud unit that is to be maintained in the read-only state, deleting the segment tree containers associated with the respective cloud unit from the local storage of the deduplicated storage appliance, and
• wherein the managing the limited metadata space on the deduplicated storage appliance further comprises detaching a particular cloud unit, the detaching comprising:
• identifying a current state of the particular cloud unit as being the read-write state, the local storage of the deduplicated storage appliance storing the segment tree containers and cloud containers associated with the particular cloud unit;
• and
• placing the particular cloud unit into the read-only state by deleting the segment tree containers associated with the particular cloud unit from the local storage of the deduplicated storage appliance, but not the cloud containers associated with the particular cloud unit.

A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method of scaling capacity in a deduplicated storage appliance having limited metadata space comprising:

• maintaining a plurality of cloud units in a cloud storage, each cloud unit comprising a plurality of containers comprising data containers storing segments of files, segment tree containers storing upper-level segments of segment trees representing the files, and cloud containers storing headers from the data and segment tree containers, wherein a header for a data container comprises fingerprints identifying the segments of the files, and a header for a segment tree container comprises fingerprints identifying the upper-level segments;
• storing, at local storage of the deduplicated storage appliance, a first level of metadata for each cloud unit in a read-write state, a second level of metadata for each cloud unit in a read-only state, and a third level of metadata for each cloud unit in an offline state, the third level of metadata requiring less storage space than the first and second levels of metadata, and the second level of metadata requiring less storage space than the first level of metadata;
• managing the limited metadata space on the deduplicated storage appliance by maintaining at least a subset of the plurality of cloud units in the read-only state;
• and
• for each cloud unit that is to be maintained in the read-only state, deleting the segment tree containers associated with the respective cloud unit from the local storage of the deduplicated storage appliance, and
• wherein the managing the limited metadata space on the deduplicated storage appliance further comprises detaching a particular cloud unit, the detaching comprising:
• identifying a current state of the particular cloud unit as being the read-only state, the local storage of the deduplicated storage appliance storing the cloud containers associated with the particular cloud unit;
• and
• placing the cloud unit into the offline state by deleting the cloud containers associated with the particular cloud unit from the local storage of the deduplicated storage appliance.