Reducing memory usage in storing metadata

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

• ingesting, at a storage system, data associated with a source system;
• and
• after the data is ingested, performing a post-processing metadata conversion process, including by:
• selecting an entry of a chunk metadata data structure;
• determining that a data chunk associated with the selected entry is not referenced by at least a threshold number of objects;
• and
• in response to determining that the data chunk associated with the selected entry is not referenced by at least the threshold number of objects:
• updating metadata of a tree data structure node corresponding to a chunk identifier associated with the data chunk to store a reference to a chunk file storing the data chunk;
• and
• removing the selected entry from the chunk metadata data structure.

A storage system, comprising:

• one or more processors configured to:
• ingest data associated with a source system;
• and
• after the data ingest is completed, perform a post-processing metadata conversion process, including by:
• selecting an entry of a chunk metadata data structure;
• determining that a data chunk associated with the selected entry is not referenced by a threshold number of objects;
• in response to determining that the data chunk associated with the selected entry is not referenced by the threshold number of objects:
• updating metadata stored by a tree data structure node corresponding to a chunk identifier associated with the data chunk to store a reference to a chunk file storing the data chunk;
• and
• removing the selected entry from the chunk metadata data structure;
• and
• a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.