Optimizing storage and retrieval of compressed data

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, program the one or more processors to perform operations comprising:

• receiving a plurality of chunks of data of a data object;
• compressing the plurality of chunks of data to obtain a plurality of compressed chunks;
• determining whether the plurality of compressed chunks together are less than a threshold size;
• based on determining the plurality of compressed chunks together are less than the threshold size, adding, to respective entries in a map data structure, chunk size representations corresponding to respective chunks of the plurality of compressed chunks, the chunk size representations in the map data structure indicating compressed sizes of respective compressed chunks of at least the plurality of compressed chunks of the data object, wherein the map data structure includes at least one offset indicator as a reference point that locates, in a compressed object, a compressed chunk associated with a subsequent entry in the map data structure that is subsequent to the at least one offset indicator, wherein a respective offset indicator of the at least one offset indicator corresponds to a respective position in the compressed object;
• compacting the map data structure to create a compacted map data structure by determining a plurality of pairs of adjacent entries in the map data structure, and, for each pair of adjacent entries of the plurality of pairs of adjacent entries in the map data structure, combining the chunk size representations of the adjacent entries in the pair of adjacent entries to create, in the compacted map data structure, a single entry including a combined chunk size representation determined by combining the chunk size representations of the adjacent entries of the pair of adjacent entries in the map data structure;
• and
• storing the plurality of compressed chunks and the compacted map data structure.

A method comprising:

• receiving, by one or more processors, a plurality of chunks of data of a data object;
• compressing the plurality of chunks of data to obtain a plurality of compressed chunks;
• determining whether the plurality of compressed chunks together are less than a threshold size;
• based on determining the plurality of compressed chunks together are less than the threshold size, adding, to respective entries in a map data structure, chunk size representations corresponding to respective chunks of the plurality of compressed chunks, the chunk size representations in the map data structure indicating compressed sizes of respective compressed chunks of at least the plurality of compressed chunks of the data object, wherein the map data structure includes at least one offset indicator as a reference point that locates, in a compressed object, a compressed chunk associated with a subsequent entry in the map data structure that is subsequent to the at least one offset indicator, wherein a respective offset indicator of the at least one offset indicator corresponds to a respective position in the compressed object;
• compacting the map data structure to create a compacted map data structure by determining a plurality of pairs of adjacent entries in the map data structure, and, for each pair of adjacent entries of the plurality of pairs of adjacent entries in the map data structure, combining the chunk size representations of the adjacent entries in the pair of adjacent entries to create, in the compacted map data structure, for each pair of adjacent entries of the plurality of pairs of adjacent entries, a single entry including a combined chunk size representation determined by combining the chunk size representations of the adjacent entries of the pair of adjacent entries in the map data structure;
• and
• storing the plurality of compressed chunks and the compacted map data structure.