Building large scale infrastructure using hybrid clusters

A method of processing jobs for clients in a distributed system, the method performed by executing computer executable instructions upon one or more computer processors, the method comprising:

• at a management cluster in a distributed system, receiving a job from a client, wherein the management cluster comprises each of a cluster manager, an OpStore, an OpStore Manager, and a job manager, the management cluster providing central monitoring of a set of one or more worker clusters which provide virtual machine environments for performing cluster jobs;
• based on the job characteristics, at the management cluster, distributing the job to a worker cluster, the worker cluster separate from the management cluster, the worker cluster being one of a plurality of worker clusters, and the worker cluster comprising one or more worker nodes;
• and
• the management cluster performing service deployment of worker nodes, service maintenance of worker nodes, service upgrade and roll-back of worker nodes, service monitoring of worker nodes, and service reporting of worker nodes for each of the plurality of worker clusters.

A distributed computing system comprising one or more computer processors upon which is executing computer executable code, the system further comprising

• a management cluster comprising:
• a cluster manager, wherein the cluster manager performs service deployment of worker nodes, service maintenance of worker nodes, and service upgrade and roll-back of worker nodes for each of one or more worker clusters;
• an OpStore, the OpStore comprising a database which stores operational and performance information for worker nodes in the distributed computing system;
• an OpStore manager;
• a job item scheduler, the job item scheduler storing jobs for worker clusters;
• a job manager coupled to the job item scheduler, wherein the job manager saves job items to the job item schedule;
• the management cluster providing central monitoring of a set of one or more worker clusters which provide virtual machine environments for performing cluster jobs, and
• wherein the management cluster receives a job from a client and distributes the job to a particular worker cluster such that the job may be performed by the particular worker cluster;
• and
• the one or more worker clusters being separate from the management cluster and coupled the management cluster, wherein each of the worker clusters comprise one or more worker nodes that provide one or more of processing power, storage, and network connectivity but do not exercise one or more management configured functions performed by the management cluster.