Scalable deduplication system with small blocks

A method for scalable data deduplication working with small data chunks, the method comprising:

• for each of the small data chunks, generating a signature using a processing device based on a combination of a representation of characters that appear in the small data chunk and a representation of frequencies of the characters in the small data chunk, wherein the signature is used to help in selecting data to be deduplicated,
• wherein the signature is generated based on a c-spectrum of the small data chunk, an f-spectrum of the small data chunk, and a p-spectrum of the small data chunk, wherein the c-spectrum of the small data chunk is a sequence of representations of different characters ordered by their frequency of occurrence in the small data chunk, the f-spectrum of the small data chunk is a corresponding sequence of frequencies of the different characters in the small data chunk, and the p-spectrum of the small data chunk is a sequence of different character pairs ordered by their frequency of occurrence in the small data chunk.