Offline deduplication for solid-state storage devices

A flash storage system comprising:

• a plurality of flash memory devices;
• a memory comprising a buffer;
• and
• a controller configured to:
• read a plurality of flash data units from the plurality of flash memory devices into the buffer, wherein each of the plurality of flash data units includes one or more host data units;
• determine an identifier for each of the host data units read into the buffer;
• select a set of unique identifiers from the determined identifiers based on a number of host data units that share the respective unique identifiers, wherein the set of unique identifiers excludes determined identifiers having a number of host data units that satisfy an upper threshold count;
• and
• for each unique identifier in the set of unique identifiers:
• designate a first host data unit sharing the unique identifier as a master data unit, wherein a logical address of the first host data unit is mapped to a first physical address in the flash memory device in a lookup table;
• remap, in the lookup table, respective logical addresses of one or more second host data units sharing the unique identifier from respective second physical addresses in the flash memory device to the first physical address in the flash memory device;
• and
• invalidate data stored at the respective second physical addresses in the flash memory device.

A non-transitory machine-readable medium comprising instructions stored therein, which when executed by a machine, cause the machine to perform operations comprising:

• reading a plurality of flash data units from flash memory into a buffer, wherein each of the plurality of flash data units includes one or more host data units;
• determining an identifier for each of the host data units read into the buffer;
• selecting a set of unique identifiers from the determined identifiers based on a number of host data units that share the respective unique identifiers, wherein the set of unique identifiers excludes determined identifiers having a number of host data units that satisfy an upper threshold count;
• and
• for each unique identifier in the set of unique identifiers:
• designating a first host data unit sharing the unique identifier as a master data unit, wherein a logical address of the first host data unit is mapped to a first physical address in the flash memory in a lookup table;
• remapping, in the lookup table, respective logical addresses of one or more second host data units sharing the unique identifier from respective second physical addresses in the flash memory to the first physical address in the flash memory;
• and
• invalidating data stored at the respective second physical addresses in the flash memory.