Small file storage system

A data processing system comprising:

• a deduplication file system having a data tree structure representing small files, the data tree structure including a data block map of any one or more segments comprising the small files, the data block map including any one of a variable block map, a fixed block map and a single block map;
• and
• a processor configured to store a file using the data block map including:
• determining whether the file is small,
• determining whether the small file should be stored using any one of the variable block map, the fixed block map and the single block map without generating metadata associated with the small file,
• generating the data block map for the small file, wherein generating the data block map for the small file includes:
• generating the fixed block map for small files in which a size of segments identified for the file range from 8 KB to 64 KB, including generating the fixed segment size for the data tree structure, generating fingerprints for each of the fixed segments comprising the small file,
• generating the variable block map for small files in which the size of segments identified for the file range from 32 KB to 64 KB, including generating the variable segment size and fingerprints for each of the variable segments comprising the file, and
• generating the single block map for small files where a size of the file is determined to be less than a threshold size for very small files of less than 10 KB, including generating the single data tree structure containing all of the data comprising the very small file,
• and
• storing the small file in an object storage based on the generated data block map without generating the metadata.

A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing small files in an object storage, the operations comprising:

• determining whether a file is small in a file system having a data tree structure representing small files, the data tree structure including a data block map of any one or more segments comprising the small files, the data block map including any one of a variable block map, a fixed block map and a single block map;
• determining whether the small file should be stored using any one of the variable block map, the fixed block map and the single block map without generating metadata associated with the small file;
• generating the data block map for the small file, wherein generating the data block map for the small file includes:
• generating the fixed block map for the small file in which the size of segments identified for the file range from 8 KB to 64 KB, including generating the fixed segment size for the data tree structure, generating fingerprints for each of the fixed segments comprising the small file,
• generating the variable block map for small files in which the size of segments identified for the file range from 32 KB to 64 KB, including generating the variable segment size and fingerprints for each of the variable segments comprising the file, and
• generating the single block map for small files where a size of the file is determined to be less than a threshold size for very small files of less than 10 KB, including generating the single data tree structure containing all of the data comprising the very small file;
• and
• storing the small file in the object storage based on the generated data block map without generating the metadata.