Dynamically downloadable distributed data deduplication library

A distributed data deduplication and retention storage system, comprising:

• a plurality of distributed clients, each hosting a software application for processing data in said system;
• and
• a plurality of distributed servers performing deduplication and storage of said data from said clients;
• and wherein each client comprises a kernel library on said each client adapted to identify an appropriate version and a location of a client library for use by said software application and to direct an API call from said application to said client library;
• said client library comprising middleware that is hosted on a server remote from said kernel library and said each client and that is formed to return to said kernel library pointers to computer code for use by said kernel library to redirect API function calls from said application to said computer program code, said computer code being formed to cooperate with said application to minimize data that is transferred from said client for deduplication and storage.

Non-transitory computer readable media embodying executable instructions for controlling the operation of a computer to perform a method of distributed processing and data deduplication for retention storage in a computer system having a client hosting an application, and having one or more servers for processing and deduplicating data from said client for said retention storage, comprising:

• receiving by a kernel library on said client an initialization API call from said application to initialize a client library for said application;
• determining by said kernel library in response to said initialization API call an appropriate version of said client library for said application and a location of said appropriate version of said client library, said determining comprising identifying said appropriate version of said client library and the location of a particular server in said computer system that hosts said appropriate version of said client library, said location being remote from said client and from said kernel library;
• returning to said kernel library by said appropriate version of said client library pointers to computer code on said particular server for redirecting API function calls from said application to said computer code on said particular server, said computer code being formed to cooperate with said application to minimize data that is transferred from said client for deduplication and storage;
• and
• redirecting by said kernel library using said pointers said API function calls from said application to said computer code for said cooperation to minimize said data transfer.