Storage management through adaptive deduplication

A system for adaptive deduplication of stored objects, comprising:

• a base deduplication module configured to perform a base deduplication process on a first portion of a plurality of objects stored on at least one storage device resulting in an estimation of a first plurality of deduplication chunk portions of the first portion of stored objects;
• a clustering module configured to categorize the first portion of the plurality of objects into deduplication sets based on a deduplication relationship of each object of the plurality of stored objects with each of the estimated first plurality of deduplication chunk portions;
• a classifier module configured to combine deduplication sets into broad classes based on deduplication characteristics of the objects in deduplication sets, build classifiers for each class using metadata attributes and content attributes of each object of the plurality of objects and classify a second portion of the plurality of objects into broad classes using the classifiers;
• and
• an adaptive deduplication module configured to select an appropriate deduplication approach for each categorized class and use a chunk portion size threshold for narrowing down chunk size for a Variable Chunk Size Hashing class.

A computer program product for adaptive deduplication of storage objects comprising a computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

• retrieve a first portion of a plurality of stored objects from at least one storage device;
• perform a base type deduplication estimation process on the first portion of stored objects resulting in an estimation of a first plurality of deduplication chunk portions of the first portion of stored objects;
• categorize the first portion of a plurality of stored objects into deduplication sets based on a deduplication relationship;
• combine deduplication sets into broad classes based on deduplication characteristics of the objects in deduplication sets;
• build classifiers for each class using metadata attributes and content attributes of the first portion of the plurality of stored objects and classify a second portion of stored objects into broad classes using the classifiers;
• and
• select an appropriate deduplication approach for each categorized class and use a chunk portion size threshold for narrowing down chunk size for a Variable Chunk Size Hashing class.