Distributed monitoring agents for cluster execution of jobs
AssigneeWorkday, Inc.
InventorsZhenqi Li, Liang Huang, Lei Gao, Hongze Lai
Life of this patent
Estimating the term…
Abstract
A system with distributed monitoring agents include a state storage, a plurality of worker agents, a first processor, and a second processor. A job is executed using a worker agent of the plurality of worker agents. The first processor is configured to execute a first monitor to monitor the job and to restart the job using job state data stored in the state storage in the event that the job fails to successfully complete. The second processor is configured to execute a second monitor to monitor the first monitor and to restart the first monitor using first monitor state data stored in the state storage in the event that the first monitor crashes.
Claims22
A system with distributed monitoring agents, comprising:
- a state storage;
- a plurality of worker agents, wherein a job is executed using a worker agent of the plurality of worker agents;
- a first processor, wherein the first processor is configured to:
- execute a first monitor to monitor the job;
- determine that the job failed to successfully complete;
- and
- in response to determining that the job failed to successfully complete:
- request to retrieve job state data stored in the state storage, wherein the job state data is retrieved using a search index comprising a location of the job state data;
- receive the job state data;
- and
- restart the job using the job state data;
- and
- a second processor, wherein the second processor is configured to:
- execute a second monitor to monitor the first monitor;
- determine that the first monitor crashes;
- and
- in response to determining that the first monitor crashes:
- restart the first monitor using first monitor state data stored in the state storage.
A method for distributed monitoring agents, comprising:
- executing a job using a worker agent, comprising a processor, of a plurality of worker agents;
- executing a first monitor to monitor the job;
- determining that the job failed to successfully complete;
- in response to determining that the job failed to successfully complete:
- requesting to retrieve job state data stored in the state storage, wherein the job state data is retrieved using a search index comprising a location of the job state data;
- receiving the job state data;
- and
- restarting the job using the job state data;
- executing a second monitor to monitor the first monitor;
- determining that the first monitor crashes;
- and
- in response to determining that the first monitor crashes:
- restarting the first monitor using first monitor state data stored in the state storage.
A computer program product for distributed monitoring agents, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- executing a job using a worker agent of a plurality of worker agents;
- executing a first monitor to monitor the job;
- determining that the job failed to successfully complete;
- in response to determining that the job failed to successfully complete:
- requesting to retrieve job state data stored in the state storage, wherein the job state data is retrieved using a search index comprising a location of the job state data;
- receiving the job state data;
- and
- restarting the job using the job state data;
- executing a second monitor to monitor the first monitor;
- determining that the first monitor crashes;
- and
- in response to determining that the first monitor crashes:
- restarting the first monitor using first monitor state data stored in the state storage.
Dependencies are inferred from the claim text.