Compression of data transmitted over a network

A network device for use in managing a data transfer over a network, comprising:

• a transceiver to send and receive the data over the network;
• and
• a computer processor executing a data reduction component that performs actions, including:
• receiving input data;
• selecting a candidate input matching data segment from within the input data based on using a sliding window over the input data to select a portion of the input data corresponding to a best fitness function value,
• wherein each fitness function value is computed from a portion of the input data within the sliding window and the best fitness function value is determined by performing a comparison between each of the plurality of computed fitness function values;
• determining a corresponding candidate store matching segment of data in a synchronized store based on an index determined from the selected candidate input matching data segment, the index being used to obtain a file identifier and offset into the synchronized store;
• selectively revising at least one boundary of the candidate input matching data segment by comparing data contiguous with the candidate store matching segment with data contiguous with the candidate input matching data segment to generate an input matching data segment of the input data;
• determining an encoded representation of the input matching data segment at least partly based on the synchronized store;
• and
• transmitting over the network a data structure that includes at least a portion of an unmatched portion of the data within the input data and the encoded representation.

A system of managing a data transfer over a network, comprising:

• a first network device that is configured to perform actions comprising:
• receiving input data;
• selecting a candidate input matching data segment from within the input data based on using a sliding window over the input data to select a portion of the input data corresponding to a best fitness function value, wherein each fitness function value is computed from a portion of the input data within the sliding window and the best fitness function value is determined by performing a comparison between each of the plurality of computed fitness function values;
• determining a corresponding candidate store matching segment of data in a synchronized store based on an index determined from the selected candidate input matching data segment, the index being used to obtain a file identifier and offset that is then used to locate within the synchronized store the candidate store matching segment;
• selectively revising at least one boundary of the candidate input matching data segment by comparing data contiguous with the candidate store matching segment with data contiguous with the candidate input matching data segment to generate an input matching data segment of the input data;
• determining an encoded representation of the input matching data segment at least partly based on the synchronized store;
• and
• transmitting over the network a data structure that includes at least a portion of an unmatched portion of the data within the input data and the encoded representation;
• and
• a second network device that is configured to perform actions comprising:
• receiving the transmitted data structure;
• employing the encoded representation in the data structure to extract, from a second synchronized store, data that matches the input matching data segment;
• and
• combining the unmatched portion of the data with the extracted data.