Moving Window Data Deduplication in Distributed Storage

A system for deduplication, comprising:

• memory storing one or more insert identifiers, each insert identifier being attached to data written to a table within a predetermined time window;
• one or more processors in communication with the memory and the table, the one or more processors configured to:
• receive a request to write data to a table, the request including an insert identifier;
• process the received insert identifier for the request when the request was received within the predetermined time window;
• determine whether table data corresponding to the received insert identifier is duplicate;
• store the received insert identifier in memory when the data is not duplicate;
• and
• update the memory to remove insert identifiers added outside of the predetermined time window.

A non-transitory computer-readable medium storing instructions executable by one or more processors for performing a method of deduplication, comprising:

• receiving a request to write data to a table, the request including an insert identifier;
• processing the received insert identifier for the request when the request was received within the predetermined time window;
• determining whether table data corresponding to the received insert identifier is duplicate;
• storing the received insert identifier in memory when the data is not duplicate;
• and
• updating the memory to remove insert identifiers added outside of the predetermined time window.