Deduplication for data transfers to portable storage devices

A method for deduplicating files during a data transfer to a data storage device, the method comprising:

• receiving a plurality of data files from a host device at a data interface of the data storage device;
• in response to transferring a first file of the plurality of data files to a data buffer of the data storage device, generating a first hash set corresponding to the first file;
• storing the first hash set in a hash buffer of the data storage device;
• storing the first file in non-volatile storage media of the data storage device;
• in response to transferring a second file of the plurality of data to the data buffer, generating a second hash set corresponding to a second file of the plurality of data files;
• in response to determining that the second hash set matches the first hash set, discarding data of the second file in the data buffer and replacing the data of the second file with a pointer to the first file;
• and
• in response to determining that the second hash does not match the first hash set or hashes for other files in the hash buffer, storing the second file in the non-volatile storage media and the second hash in the hash buffer.

A data storage device comprising:

• means for receiving power and data from a host device;
• a data buffer;
• a hash buffer;
• means for permanently storing data;
• means for generating hash sets;
• control circuitry configured to:
• receive a plurality of data files from the host device via the means for receiving power and data;
• in response to transferring a first file of the plurality of data files to the data buffer, obtaining a first hash set corresponding to the first file from the means for generating hash sets;
• store the first hash set in the hash buffer;
• store the first file in the means for permanently storing data;
• in response to transferring a second file of the plurality of data files to the data buffer, obtaining a second hash set corresponding to a second file of the plurality of data files from the means for generating hash sets;
• in response to determining that the second hash set matches the first hash set, discard data of the second file in the data buffer and replace the data of the second file with a pointer to the first file;
• and
• in response to determining that the second hash does not match the first hash set or hashes for other files in the hash buffer, store the second file in the means for permanently storing data and the second hash in the hash buffer.