Object format and upload process for archiving data in cloud/object storage

A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code embodying a method for archiving a data set in cloud storage, the method comprising:

• receiving an incremental snapshot of the data set;
• packaging the incremental snapshot into one or more data objects and one or more metadata objects, wherein the one or more data objects include data blocks of the incremental snapshot, and wherein the one or more metadata objects include a metadata entry for each data block, the metadata entry comprising an identifier of the incremental snapshot, an identifier of the data block, and an identifier of a data object in which the data block is stored;
• and
• writing the one or more data objects and the one or more metadata objects to the cloud storage,
• wherein the packaging and writing comprises, as each data block of the incremental snapshot is received by the computer system:
• adding the data block to a fixed-size data memory buffer;
• adding a metadata entry for the data block to a metadata memory buffer;
• determining whether the data memory buffer is full upon adding the data block;
• and
• if the data memory buffer is determined to be full, writing contents of the data memory buffer to the cloud storage as a new data object, and
• wherein once all data blocks of the incremental snapshot are processed via the data memory buffer, the computer system writes contents of the metadata memory buffer to the cloud storage as a new metadata object.

A computer system comprising:

• a processor;
• and
• a non-transitory computer readable medium having stored thereon program code for archiving a data set in cloud storage, the program code causing the processor to:
• receive an incremental snapshot of the data set;
• package the incremental snapshot into one or more data objects and one or more metadata objects, wherein the one or more data objects include data blocks of the incremental snapshot, and wherein the one or more metadata objects include a metadata entry for each data block, the metadata entry comprising an identifier of the incremental snapshot, an identifier of the data block, and an identifier of a data object in which the data block is stored;
• and
• write the one or more data objects and the one or more metadata objects to the cloud storage,
• wherein the program code that causes the processor to package incremental snapshot and write the one or more data objects comprises program code that causes the processor to, as each data block of the incremental snapshot is received:
• add the data block to a fixed-size data memory buffer;
• add a metadata entry for the data block to a metadata memory buffer;
• determine whether the data memory buffer is full upon adding the data block;
• and
• if the data memory buffer is determined to be full, write contents of the data memory buffer to the cloud storage as a new data object, and
• wherein once all data blocks of the incremental snapshot are processed via the data memory buffer, the processor writes contents of the metadata memory buffer to the cloud storage as a new metadata object.