Systems and methods for sequence-based data chunking for deduplication

A method for data chunking, the method comprising:

• identifying, by a hardware processor, a desired throughput value for data chunking a byte stream of data by:
• determining, by the hardware processor, a size of the byte stream;
• receiving, by the hardware processor, a threshold period of time to complete the data chunking;
• and
• calculating, by the hardware processor, the desired throughput value based on the size of the byte stream and the threshold period of time;
• setting, by the hardware processor, a first amount of adjacent bytes, a second amount of bytes, and a fixed amount of bytes of a jump mechanism based on the desired throughput value;
• consecutively scanning, by the hardware processor, each respective byte in the byte stream of data;
• in response to detecting the first amount of adjacent bytes with values arranged in an increasing order, marking, by the hardware processor, a cut-off point of a data chunk comprising at least the adjacent bytes;
• in response to detecting the second amount of bytes with values arranged in a decreasing order, executing, by the hardware processor, the jump mechanism that skips scanning of the fixed amount of bytes after the bytes with values arranged in the decreasing order;
• and
• subsequent to scanning the byte stream in entirety, outputting, by the hardware processor, a plurality of cut-off points of identified data chunks.

A system for data chunking, comprising:

• at least one memory;
• and
• at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
• identify a desired throughput value for data chunking a byte stream of data by:
• determining a size of the byte stream;
• receiving a threshold period of time to complete the data chunking;
• and
• calculating the desired throughput value based on the size of the byte stream and the threshold period of time;
• set a first amount of adjacent bytes, a second amount of bytes, and a fixed amount of bytes of a jump mechanism that achieve the desired throughput value;
• consecutively scan each respective byte in the byte stream of data;
• in response to detecting the first amount of adjacent bytes with values arranged in an increasing order, mark a cut-off point of a data chunk comprising at least the adjacent bytes;
• in response to detecting the second amount of bytes with values arranged in a decreasing order, execute the jump mechanism that skips scanning of the fixed amount of bytes after the bytes with values arranged in the decreasing order;
• and
• subsequent to scanning the byte stream in entirety, output a plurality of cut-off points of identified data chunks.