Method and system for scalable job processing
AssigneeImport.Io Limited
InventorsMatthew James George Painter, Ian Andrew Clark
Life of this patent
Estimating the term…
Abstract
The present invention relates to methods for processing jobs within a cluster architecture. One method comprises the pausing of a job when waiting upon external dependencies. Another method comprises the transmission of messages relating to the ongoing processing of jobs back to a client via a persistent messaging channel. Yet another method comprises determining capacity at a node before allocating a job for processing by the node or adding the job to a cluster queue. A system for processing jobs within a cluster architecture is also disclosed.
Claims19
A method for processing jobs in a cluster architecture including a cluster queue for providing jobs to a plurality of processing nodes, each node of the plurality of processing nodes including a local queue for providing jobs, the method including:
- one node of a plurality of processing nodes within the cluster architecture receiving a job;
- processing the job until the job is waiting for a dependency external to the cluster architecture to complete;
- when the job is waiting for the external dependency, pausing the job by the node;
- after pausing the job, retrieving another job from a local queue for the node if the local queue includes another job and from the cluster queue if the local queue does not include another job;
- processing the retrieved another job by the node;
- and
- when the external dependency is completed, allocating the paused job to the local queue for the node to continue the job;
- wherein the job is received from an external client and messages in relation to the job are delivered back to the external client via a persistent messaging channel;
- further comprising determining, by the node, whether to process the received job locally or to push the received job to the cluster queue based on a current and/or projected processing load of the node;
- and processing the job by the node based on determining to process the received job locally.
A system comprising:
- a plurality of processing nodes within a cluster architecture including a cluster queue for providing jobs to the plurality of processing nodes, each node of the plurality of processing nodes including a local queue for providing jobs;
- and
- a communications system;
- wherein the system is configured to perform a method for processing jobs in the cluster architecture, the method comprising:
- one node of a plurality of processing nodes within the cluster architecture receiving a job;
- processing the job until the job is waiting for a dependency external to the cluster architecture to complete;
- when the job is waiting for the external dependency, pausing the job by the node;
- after pausing the job, retrieving another job from a local queue for the node if the local queue includes another job and from the cluster queue if the local queue does not include another job;
- processing the received another job by the node;
- and
- when the external dependency is completed, allocating the paused job to the local queue for the node to continue the job;
- wherein the job is received from an external client and messages in relation to the job are delivered back to the external client via a persistent messaging channel;
- further comprising determining, by the node, whether to process the received job locally or to push the received job to the cluster queue based on a current and/or projected processing load of the node;
- and processing the job by the node based on determining to process the received job locally.
A non-transitory computer readable storage medium having stored therein computer instructions which, when executed by a processor of a node of a plurality of processing nodes, cause the node to perform operations for processing jobs in a cluster architecture including a cluster queue for providing jobs to the plurality of processing nodes, each node of the plurality of processing nodes including a local queue for providing jobs, the operations comprising:
- one node of the plurality of processing nodes within the cluster architecture receiving a job;
- processing the job until the job is waiting for a dependency external to the cluster architecture to complete;
- when the job is waiting for the external dependency, pausing the job by the node;
- after pausing the job, retrieving another job from a local queue for the node if the local queue includes another job and from the cluster queue if the local queue does not include another job;
- processing the retrieved another job by the node;
- and
- when the external dependency is completed, allocating the paused job to the local queue for the node to continue the job;
- wherein the job is received from an external client and messages in relation to the job are delivered back to the external client via a persistent messaging channel;
- further comprising determining, by the node, whether to process the received job locally or to push the received job to the cluster queue based on a current and/or projected processing load of the node;
- and processing the job by the node based on determining to process the received job locally.
Dependencies are inferred from the claim text.