System and method for efficient storage and version control of machine learning datasets using hybrid delta encoding

A method for efficient storage and version control of machine learning datasets that reduces storage usage and reconstruction latency, comprising:

• storing dataset information in a data storage layer in a columnar format comprising a plurality of columns, each column corresponding to a chain of data blocks comprising at least one full-data block and at least one delta-encoded block, the at least one delta-encoded block comprising operations to modify a preceding block comprising at least one of at least one keep operation, at least one delete operation, at least one update operation, and at least one insert operation;
• storing version metadata in a header storage layer, the version metadata comprising a plurality of content-addressable header files, at least one header file representing a dataset version and comprising column metadata mapping at least one column to a head block in the chain of data blocks;
• generating, in response to a commit request comprising changes relative to a prior dataset version, at least one new delta-encoded block in the chain of data blocks for at least one column identified as modified and a new header file representing a new dataset version, the new header file comprising column metadata mapping the at least one column to a head block in the chain of data blocks;
• and
• in response to a query for a requested dataset version, reconstructing data for at least one column for a requested dataset version by identifying a header file corresponding to the requested dataset version and processing delta-encoded blocks in the chain of data blocks for the at least one column relative to a full-data block.

A non-transitory, computer-readable medium comprising instructions that, when executed by at least one processing resource, cause the at least one processing resource to:

• store dataset information in a data storage layer in a columnar format comprising a plurality of columns, each column corresponding to a chain of data blocks comprising at least one full-data block and at least one delta-encoded block, the at least one delta-encoded block comprising operations to modify a preceding block comprising at least one of at least one keep operation, at least one delete operation, at least one update operation, and at least one insert operation;
• store version metadata in a header storage layer, the version metadata comprising a plurality of content-addressable header files, at least one header file representing a dataset version and comprising column metadata mapping at least one column to a head block in the chain of data blocks;
• generate, in response to a commit request comprising changes relative to a prior dataset version, at least one new delta-encoded block in the chain of data blocks for at least one column identified as modified and a new header file representing a new dataset version, the new header file comprising column metadata mapping the at least one column to a head block in the chain of data blocks;
• and
• in response to a query for a requested dataset version, reconstruct data for at least one column for a requested dataset version by identifying a header file corresponding to the requested dataset version and processing delta-encoded blocks in the chain of data blocks for the at least one column relative to a full-data block.