Mechanism for facilitating a quorum-based coordination of broker health for management of resources for application servers in an on-demand services environment

A database system comprising:

• a health-checker server device having a memory to store instructions, and a processing device to execute the instructions, the health-checker server device to perform operations comprising:
• monitoring resource distribution health of message queue brokers managing resource domains having worker nodes and thread resources for processing job requests, wherein managing includes routing of the job requests to job queues associated with application servers in a multi-tenant resource distribution environment and in communication over a network, wherein each message queue broker is associated with at least one resource domain having a set of working nodes and a set of thread resources;
• generating, by the health checker of the database system, a status report associated with a first message queue broker based on quorum-based votes casted by other message queue brokers, wherein the status report to indicate fair or unfair management of a first resource domain by the first message queue broker, wherein each message queue broker participates in a quorum-based voting system to vote to rate one or more of the message queue brokers on their management of one or more of the resource domains;
• and
• classifying, based on the quorum-based votes, the first message queue broker as one of successful, failed and recoverable, and failed and unrecoverable,
• wherein the first message queue broker when classified as successful, the first message queue broker remains active such that a set of job requests distributed to the first resource domain associated with the first message queue broker is routed to one or more of the job queues without interruption
• wherein the first message queue broker when classified as failed and recoverable, the first message queue broker is recovered from failure and turned active to resume routing of the set of job requests, and
• wherein the first message queue broker when classified as failed and unrecoverable, the first message queue broker is retired, and the set of job requests is distributed to a second message queue broker managing a second resource domain.

At least one non-transitory machine-readable medium having stored thereon instructions which, when executed by a health-checker processing device, causes the health-checker processing device to perform operations comprising:

• monitoring resource distribution health of message queue brokers managing resource domains having worker nodes and thread resources for processing job requests, wherein managing includes routing of the job requests to job queues associated with application servers in a multi-tenant resource distribution environment and in communication over a network, wherein each message queue broker is associated with at least one resource domain having a set of working nodes and a set of thread resources;
• generating, by the health checker of the database system, a status report associated with a first message queue broker based on quorum-based votes casted by other message queue brokers, wherein the status report to indicate fair or unfair management of a first resource domain by the first message queue broker, wherein each message queue broker participates in a quorum-based voting system to vote to rate one or more of the message queue brokers on their management of one or more of the resource domains;
• and
• classifying, based on the quorum-based votes, the first message queue broker as one of successful, failed and recoverable, and failed and unrecoverable,
• wherein the first message queue broker when classified as successful, the first message queue broker remains active such that a set of job requests distributed to the first resource domain associated with the first message queue broker is routed to one or more of the job queues without interruption,
• wherein the first message queue broker when classified as failed and recoverable, the first message queue broker is recovered from failure and turned active to resume routing of the set of job requests, and
• wherein the first message queue broker when classified as failed and unrecoverable, the first message queue broker is retired, and the set of job requests is distributed to a second message queue broker managing a second resource domain.