Managing data transfer between endpoints in a distributed computing environment

A method for providing a file fetcher arranged for serving and retrieving files to and from endpoints on a network supporting peer-to-peer file transfer, the method comprising the steps of:

• associating work items in a work list with pending file requests from a caller at an endpoint, the work list being arranged to support prioritization of the work items;
• obtaining a list of endpoints that contain at least a portion of files associated with the pending file requests;
• maintaining a state for a file, the state identifying the file as being partially or completely downloaded;
• retrieving at least a portion of a file in accordance with the work list, the list of endpoints, and the file state;
• and
• serving at least a portion of a file responsively to a request from a caller at a remote endpoint,
• wherein said retrieving step further comprises downloading the file using chunked file fetching and includes the further steps of:
• receiving a list of available file chunks at an endpoint, and
• receiving, from the caller, a number of file chunks and associated size of each chunk for creating the file state for a file, wherein space of a required size is allocated, at the file fetcher, prior to retrieving the file, based upon said number of file chunks and associated size of each chunk received from the caller, and
• wherein the work list contains a plurality of work items for a file so that the file is retrieved in chunks in parallel from a plurality of endpoints;
• in which a bitmap is appended at the end of the file and is maintained in an in-memory cache;
• and
• wherein the bitmap appended to the file is removed after a last chunk is written to the file.