Method for intercepting an instruction produced by an application on a computer

A distributed computing system comprising:

• a plurality of compute nodes assigned to execute one or more tasks corresponding to a job assigned to the plurality of compute nodes, each compute node comprising local storage;
• remote storage accessible by each compute node;
• each compute node comprising a middleware layer that permits each compute node to interface with a given task processor regardless of the compute node's underlying configuration, and the middleware layer comprising a file system interception layer that does not require downloading all file dependencies otherwise required for processing the one or more tasks assigned to a compute node by automatically downloading any missing data files to local storage of a compute node processing a task as and when such missing data files are needed to process task dependent files, wherein downloading any missing data files at the file system interception layer comprises:
• detecting a request to retrieve a data file from a specified path on local storage of the compute node;
• temporarily suspending the request from completion;
• checking to determine whether the requested data file is available on the compute node's local storage;
• and
• if the requested data file is available, allowing the request to proceed to completion;
• and
• if the requested data file is not available, downloading the requested data file from remote storage of the distributed computing system to the specified path on the local storage of the compute node, and wherein the requested data file is stored on the remote storage with a file hierarchy that is the same hierarchy as would be used if the requested data file were stored on the compute node's local storage.

A computer program product comprising a hardware storage device containing computer-executable instructions which, when executed by one or more processors, causes execution of a computer-implemented method comprising:

• assigning a plurality of compute nodes of a distributed computing system to execute one or more tasks corresponding to a job assigned to the plurality of compute nodes;
• provisioning each compute node with a middleware layer that permits each compute node to interface with a given task processor regardless of the compute node's underlying configuration, and the middleware layer comprising a file system interception layer;
• wherein the file system interception layer automatically downloads any missing data files to local storage of the compute node as and when any missing data files are needed to process task dependent files so that the automatic download does not download file dependencies that are not required in order to process the one or more tasks assigned to a compute node, and wherein automatically downloading any missing data files at the file system interception layer comprises:
• detecting a request to retrieve a data file from a specified path on local storage of the compute node;
• temporarily suspending the request from completion;
• checking to determine whether the requested data file is available on the compute node's local storage;
• and
• if the requested data file is available, allowing the request to proceed to completion;
• and
• if the requested data file is not available, downloading the requested data file from remote storage of the distributed computing system to the specified path on the local storage of the compute node, and wherein the requested data file is stored on the remote storage with a file hierarchy that is the same hierarchy as would be used if the requested data file were stored on the computer node's local storage.