Data transfer acceleration via content-defined chunking

One or more tangible, non-transitory computer-readable media having stored thereon instructions, the instructions when executed to instruct one or more processors to:

• compute an output of a first simulation task using a first filesystem image, the first filesystem image having a data structure that includes autonomous vehicle sensor data for autonomous vehicle simulation;
• cache the first filesystem image;
• use content-defined chunking (CDC) to divide the first filesystem image into a plurality of variable-sized chunks;
• compute individual hashes for the variable-sized chunks, and store the individual hashes in a memory;
• download a second filesystem image from a remote data source, wherein the second filesystem image is used for a second simulation task and divided into a plurality of variable-sized chunks, wherein downloading the second filesystem image comprises receiving hashes for the variable-sized chunks of the second filesystem image, and querying the memory for matching individual hashes of the variable-sized chunks of the first filesystem image, and downloading chunks corresponding to the matching individual hashes;
• build a copy of the second filesystem image from a combination of the downloaded chunks and the matching chunks found in the querying;
• and
• compute an output of the second simulation task using the copy of the second filesystem image.

A computer-implemented method, comprising:

• performing, at a worker node, a first simulation task using a first binary image, the first binary image having a data structure that includes autonomous vehicle sensor data for autonomous vehicle simulation;
• dividing the first binary image into a plurality of variable-sized chunks, wherein the first binary image is an aggregate of a plurality of files, and wherein the dividing does not depend on file boundaries;
• computing hashes of the variable-sized chunks, and storing the hashes in a content addressable storage (CAS) with the hashes as keys;
• receiving a manifest comprising an ordered set of second hashes, wherein the second hashes are hashes of chunks of a second binary image to be downloaded and used in a second simulation task;
• building a download plan, comprising searching the CAS for the second hashes, compiling a first group of hashes found in the CAS, and compiling a second group of hashes not found in the CAS, and ordering the first group of hashes and second set of hashes to comply with the manifest;
• downloading chunks corresponding to the second group of hashes;
• reordering the downloaded chunks and chunks that correspond to first group of hashes to build a copy of the second binary image;
• and
• performing, at the worker node, the second simulation task using the copy of the second binary image.